[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:07 dist-test-slave-6bbx krb5kdc[15596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:07 dist-test-slave-6bbx krb5kdc[15596](info): set up 2 sockets
Aug 10 23:01:07 dist-test-slave-6bbx krb5kdc[15596](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 23:01:08.824501 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:08 dist-test-slave-6bbx krb5kdc[15596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:08.960162 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33897
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:33897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:09.077543 15617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:09.077848 15617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:09.077908 15617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:09.083552 15617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:09.083649 15617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:09.083714 15617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:09.083750 15617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:09.083814 15617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:09.089706 15617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:33897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:09.091152 15617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:09.092383 15617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:09.101823 15624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:09.101864 15625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:09.101822 15627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:09.102254 15617 server_base.cc:1034] running on GCE node
I20260810 23:01:09.102959 15617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:09.104259 15617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:09.105424 15617 hybrid_clock.cc:648] HybridClock initialized: now 1786402869105434 us; error 71 us; skew 500 ppm
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:09.108417 15617 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:09.109421 15617 webserver.cc:460] Webserver started at http://127.15.56.62:40287/ using document root <none> and password file <none>
I20260810 23:01:09.110037 15617 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:09.110108 15617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:09.110332 15617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:09.112637 15617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "0aa74408fdba4f309a711c700b6ba97d"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "9aa5a2e4c6a9557517a0522ec98cadbb"
server_key_iv: "e37201199c795d26d0cbb4b436a3a11e"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.113217 15617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "0aa74408fdba4f309a711c700b6ba97d"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "9aa5a2e4c6a9557517a0522ec98cadbb"
server_key_iv: "e37201199c795d26d0cbb4b436a3a11e"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.117177 15617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:01:09.119891 15640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:09.121227 15617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:09.121362 15617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "0aa74408fdba4f309a711c700b6ba97d"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "9aa5a2e4c6a9557517a0522ec98cadbb"
server_key_iv: "e37201199c795d26d0cbb4b436a3a11e"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.121471 15617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:09.134294 15617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:09.137557 15617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:09.137773 15617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:09.145175 15617 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:33897
I20260810 23:01:09.145212 15726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:33897 every 8 connection(s)
I20260810 23:01:09.146260 15617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:09.148360 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 15617
I20260810 23:01:09.148519 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:09.149578 15727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:09.149917 15584 external_mini_cluster.cc:1428] Setting key b08f88ceec837f5f3d8a7804e3a68791
I20260810 23:01:09.155923 15727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d: Bootstrap starting.
I20260810 23:01:09.158538 15727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:09.159518 15727 log.cc:824] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:09.161818 15727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d: No bootstrap required, opened a new log
I20260810 23:01:09.166117 15727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa74408fdba4f309a711c700b6ba97d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33897 } }
I20260810 23:01:09.166366 15727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:09.166404 15727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa74408fdba4f309a711c700b6ba97d, State: Initialized, Role: FOLLOWER
I20260810 23:01:09.166925 15727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa74408fdba4f309a711c700b6ba97d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33897 } }
I20260810 23:01:09.167092 15727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:09.167166 15727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:09.167266 15727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:09.168532 15727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa74408fdba4f309a711c700b6ba97d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33897 } }
I20260810 23:01:09.168962 15727 leader_election.cc:302] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aa74408fdba4f309a711c700b6ba97d; no voters:
I20260810 23:01:09.169360 15727 leader_election.cc:288] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [CANDIDATE]: Term 1 election: Requested vote from peers
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:09.169874 15739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:09.170230 15739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [term 1 LEADER]: Becoming Leader. State: Replica: 0aa74408fdba4f309a711c700b6ba97d, State: Running, Role: LEADER
I20260810 23:01:09.170660 15739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa74408fdba4f309a711c700b6ba97d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33897 } }
I20260810 23:01:09.170943 15727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:09.172537 15741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aa74408fdba4f309a711c700b6ba97d. Latest consensus state: current_term: 1 leader_uuid: "0aa74408fdba4f309a711c700b6ba97d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa74408fdba4f309a711c700b6ba97d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33897 } } }
I20260810 23:01:09.172683 15741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:09.173024 15738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aa74408fdba4f309a711c700b6ba97d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aa74408fdba4f309a711c700b6ba97d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33897 } } }
I20260810 23:01:09.173146 15738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d [sys.catalog]: This master's current role is: LEADER
W20260810 23:01:09.176685 15763 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:01:09.176868 15763 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:01:09.176993 15751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:09.178025 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:09.153364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.15.56.62:33897)
0810 23:01:09.153927 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:09.153948 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:09.155039 (+ 1091us) server_negotiation.cc:407] Connection header received
0810 23:01:09.158437 (+ 3398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:09.158462 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:09.158931 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:09.159308 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:09.161824 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.163292 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.164645 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.165404 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.165581 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:09.171045 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:09.171053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:09.171071 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:09.171113 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:09.173454 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.174205 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.174212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.174218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.174384 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.174835 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.174840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.174841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.175392 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:09.175613 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:09.176422 (+ 809us) server_negotiation.cc:299] Negotiation successful
0810 23:01:09.176756 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":157,"threads_started":1}
I20260810 23:01:09.178103 15751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:09.184617 15751 catalog_manager.cc:1347] Generated new cluster ID: cc3493e6ed704092b1d0ba7f956739c5
I20260810 23:01:09.184697 15751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:09.196358 15751 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:09.197734 15751 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:09.203871 15751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aa74408fdba4f309a711c700b6ba97d: Generated new TSK 0
I20260810 23:01:09.204829 15751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:01:09.249221 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33897
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:09.377535 15774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:09.378010 15774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:09.378110 15774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:09.382450 15774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:09.382539 15774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:09.382603 15774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:09.388413 15774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:09.389716 15774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:09.390916 15774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:09.401182 15781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:09.401182 15786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:09.401216 15784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:09.401381 15774 server_base.cc:1034] running on GCE node
I20260810 23:01:09.401986 15774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:09.402712 15774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:09.403878 15774 hybrid_clock.cc:648] HybridClock initialized: now 1786402869403854 us; error 33 us; skew 500 ppm
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:09.406810 15774 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:01:09.407816 15774 webserver.cc:460] Webserver started at http://127.15.56.1:36331/ using document root <none> and password file <none>
I20260810 23:01:09.408385 15774 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:09.408445 15774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:09.408631 15774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:09.410851 15774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "8f51b8ba96d54eea86526646be97cb4f"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "352069591097cb5763ca466f64cfa7d7"
server_key_iv: "8dc05a1c75e3d5d99ddba2acd529da03"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.411516 15774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "8f51b8ba96d54eea86526646be97cb4f"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "352069591097cb5763ca466f64cfa7d7"
server_key_iv: "8dc05a1c75e3d5d99ddba2acd529da03"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.415937 15774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 23:01:09.419078 15795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:09.420639 15774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:09.420795 15774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "8f51b8ba96d54eea86526646be97cb4f"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "352069591097cb5763ca466f64cfa7d7"
server_key_iv: "8dc05a1c75e3d5d99ddba2acd529da03"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.420919 15774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:09.435742 15774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:09.439370 15774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:09.439585 15774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:09.440266 15774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:09.441185 15774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:09.441238 15774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:09.441296 15774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:09.441321 15774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:09.449882 15774 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:45909
I20260810 23:01:09.449950 15965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:45909 every 8 connection(s)
I20260810 23:01:09.450888 15774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:09.458258 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 15774
I20260810 23:01:09.458379 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:09.458623 15584 external_mini_cluster.cc:1428] Setting key 1f0a43733abde17d49e06c454ee58dfd
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:09.466348 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:09.453008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:49117 (local address 127.15.56.62:33897)
0810 23:01:09.453182 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:09.453186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:09.454341 (+ 1155us) server_negotiation.cc:407] Connection header received
0810 23:01:09.455300 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:09.455304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:09.455364 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:09.455442 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:09.456809 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.457362 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.458593 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.458881 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.458974 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:09.461997 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:09.462002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:09.462004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:09.462029 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:09.463751 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.464468 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.464471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.464472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.464565 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.464977 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.464980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.464981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.465185 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:09.465293 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:09.466051 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 23:01:09.466213 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:01:09.467646 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:09.453451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33897 (local address 127.15.56.1:49117)
0810 23:01:09.454158 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:09.454216 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:09.455069 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:09.455642 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:09.455650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:09.456085 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:09.456644 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:09.456660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.457494 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.457497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:09.458466 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:09.458478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.459053 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.459057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:09.459266 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:09.459989 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:09.460010 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:09.461808 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:09.463912 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:09.463920 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:09.463941 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:09.464343 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:09.464660 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:09.464663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:09.464664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:09.464875 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:09.465324 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:09.465332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:09.465741 (+ 409us) client_negotiation.cc:769] Sending connection context
0810 23:01:09.466003 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 23:01:09.466426 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":126,"threads_started":1}
I20260810 23:01:09.469084 15966 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33897
I20260810 23:01:09.469373 15966 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:09.469929 15966 heartbeater.cc:511] Master 127.15.56.62:33897 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:01:09.471798 15669 ts_manager.cc:194] Registered new tserver with Master: 8f51b8ba96d54eea86526646be97cb4f (127.15.56.1:45909)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
I20260810 23:01:09.473168 15669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:49117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:01:09.491544 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33897
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:09.608826 15978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:09.609110 15978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:09.609156 15978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:09.613212 15978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:09.613299 15978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:09.613364 15978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:09.618958 15978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:09.620143 15978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:09.621214 15978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:09.631174 15985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:09.631220 15978 server_base.cc:1034] running on GCE node
W20260810 23:01:09.631167 15990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:09.631167 15994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:09.631888 15978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:09.632550 15978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:09.633713 15978 hybrid_clock.cc:648] HybridClock initialized: now 1786402869633689 us; error 32 us; skew 500 ppm
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:09.636469 15978 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:01:09.637363 15978 webserver.cc:460] Webserver started at http://127.15.56.2:46115/ using document root <none> and password file <none>
I20260810 23:01:09.637914 15978 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:09.637985 15978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:09.638206 15978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:09.640370 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "82c3e92e3b944905a150903fc1cf425a"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "0211b94f337c4ad18c6254993b666afa"
server_key_iv: "23d3b218994fb15c3248447ddbbc8382"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.640926 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "82c3e92e3b944905a150903fc1cf425a"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "0211b94f337c4ad18c6254993b666afa"
server_key_iv: "23d3b218994fb15c3248447ddbbc8382"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.644917 15978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 23:01:09.647890 16005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:09.649411 15978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:01:09.649554 15978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "82c3e92e3b944905a150903fc1cf425a"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "0211b94f337c4ad18c6254993b666afa"
server_key_iv: "23d3b218994fb15c3248447ddbbc8382"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.649683 15978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:09.671175 15978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:09.674840 15978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:09.675046 15978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:09.675729 15978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:09.676635 15978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:09.676687 15978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:09.676743 15978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:09.676772 15978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:09.685735 15978 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:42191
I20260810 23:01:09.685760 16181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:42191 every 8 connection(s)
I20260810 23:01:09.686717 15978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:09.689397 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 15978
I20260810 23:01:09.689508 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:09.689793 15584 external_mini_cluster.cc:1428] Setting key 283b9365195660fba6487eb3114c40d0
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:01:09.701041 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:09.688650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:49717 (local address 127.15.56.62:33897)
0810 23:01:09.688822 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:09.688827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:09.690012 (+ 1185us) server_negotiation.cc:407] Connection header received
0810 23:01:09.691095 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:09.691098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:09.691146 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:09.691219 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:09.692535 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.693172 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.694327 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.694548 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.694632 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:09.697345 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:09.697348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:09.697350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:09.697376 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:09.698767 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.699364 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.699366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.699368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.699440 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.699868 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.699870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.699871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.700053 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:09.700130 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:09.700805 (+ 675us) server_negotiation.cc:299] Negotiation successful
0810 23:01:09.700915 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:01:09.702075 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:09.689081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33897 (local address 127.15.56.2:49717)
0810 23:01:09.689859 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:09.689905 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:09.690869 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:09.691411 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:09.691418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:09.691862 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:09.692392 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:09.692407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.693307 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.693310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:09.694200 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:09.694210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.694666 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.694669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:09.694827 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:09.695381 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:09.695398 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:09.697172 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:09.698897 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:09.698904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:09.698922 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:09.699274 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:09.699527 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:09.699530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:09.699532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:09.699764 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:09.700162 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:09.700169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:09.700524 (+ 355us) client_negotiation.cc:769] Sending connection context
0810 23:01:09.700751 (+ 227us) client_negotiation.cc:240] Negotiation successful
0810 23:01:09.701067 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
I20260810 23:01:09.703327 16187 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33897
I20260810 23:01:09.703613 16187 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:09.704284 16187 heartbeater.cc:511] Master 127.15.56.62:33897 requested a full tablet report, sending...
I20260810 23:01:09.705461 15669 ts_manager.cc:194] Registered new tserver with Master: 82c3e92e3b944905a150903fc1cf425a (127.15.56.2:42191)
I20260810 23:01:09.706082 15669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:49717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:01:09.719094 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33897
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:09.836819 16193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:09.837101 16193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:09.837152 16193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:09.841207 16193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:09.841274 16193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:09.841348 16193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:09.846815 16193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33897
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:09.848035 16193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:09.849128 16193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:09.859253 16207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:09.859265 16201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:09.859252 16203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:09.859532 16193 server_base.cc:1034] running on GCE node
I20260810 23:01:09.860077 16193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:09.860805 16193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:09.861970 16193 hybrid_clock.cc:648] HybridClock initialized: now 1786402869861951 us; error 32 us; skew 500 ppm
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:09.864912 16193 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:01:09.865880 16193 webserver.cc:460] Webserver started at http://127.15.56.3:46211/ using document root <none> and password file <none>
I20260810 23:01:09.866479 16193 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:09.866545 16193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:09.866770 16193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:09.869028 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "ac53e6c620aa4b6eb5de29eea70c802b"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "1c384dc11f90d61ce8c89bc72bc09c0d"
server_key_iv: "86d0f857d37763205ca57ca70a1de861"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.869632 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "ac53e6c620aa4b6eb5de29eea70c802b"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "1c384dc11f90d61ce8c89bc72bc09c0d"
server_key_iv: "86d0f857d37763205ca57ca70a1de861"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.873740 16193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:01:09.876561 16217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:09.877849 16193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:09.877959 16193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "ac53e6c620aa4b6eb5de29eea70c802b"
format_stamp: "Formatted at 2026-08-10 23:01:09 on dist-test-slave-6bbx"
server_key: "1c384dc11f90d61ce8c89bc72bc09c0d"
server_key_iv: "86d0f857d37763205ca57ca70a1de861"
server_key_version: "encryptionkey@0"
I20260810 23:01:09.878079 16193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:09.893175 16193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:09.896912 16193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:09.897113 16193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:09.897804 16193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:09.898720 16193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:09.898780 16193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:09.898824 16193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:09.898854 16193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:09.907969 16193 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:33381
I20260810 23:01:09.907996 16395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:33381 every 8 connection(s)
I20260810 23:01:09.908917 16193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:09.916426 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 16193
I20260810 23:01:09.916536 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:09.916805 15584 external_mini_cluster.cc:1428] Setting key 361267eb35bafc36c2e2b1ed01eab627
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402869, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:09.922996 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:09.910956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:56125 (local address 127.15.56.62:33897)
0810 23:01:09.911081 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:09.911084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:09.912237 (+ 1153us) server_negotiation.cc:407] Connection header received
0810 23:01:09.913147 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:09.913151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:09.913212 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:09.913287 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:09.914536 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.915073 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.916360 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.916641 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.916740 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:09.919712 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:09.919716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:09.919719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:09.919749 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:09.920966 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.921465 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.921468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.921470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.921559 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.921883 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.921886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.921887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.922086 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:09.922217 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:09.922740 (+ 523us) server_negotiation.cc:299] Negotiation successful
0810 23:01:09.922864 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:01:09.923899 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:09.911337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33897 (local address 127.15.56.3:56125)
0810 23:01:09.912061 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:09.912104 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:09.912935 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:09.913469 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:09.913476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:09.913858 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:09.914387 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:09.914402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.915200 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.915203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:09.916178 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:09.916190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.916815 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.916819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:09.917016 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:09.917632 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:09.917649 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:09.919518 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:09.921074 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:09.921079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:09.921093 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:09.921358 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:09.921636 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:09.921638 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:09.921639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:09.921779 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:09.922212 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:09.922218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:09.922476 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 23:01:09.922638 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 23:01:09.922940 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":128,"threads_started":1}
I20260810 23:01:09.925596 16396 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33897
I20260810 23:01:09.925922 16396 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:09.926576 16396 heartbeater.cc:511] Master 127.15.56.62:33897 requested a full tablet report, sending...
I20260810 23:01:09.928048 15669 ts_manager.cc:194] Registered new tserver with Master: ac53e6c620aa4b6eb5de29eea70c802b (127.15.56.3:33381)
I20260810 23:01:09.928695 15669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:56125
I20260810 23:01:09.931785 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402869, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402869, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:01:09.958935 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:09.949685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.15.56.1:45909)
0810 23:01:09.949998 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:09.950013 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:09.950039 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:01:09.950129 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:09.950135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:09.950341 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:09.950466 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:09.951082 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.951930 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.952772 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.953171 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.953271 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:09.955625 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:09.955635 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:09.955651 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:09.955713 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:09.957214 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.957604 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.957609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.957610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.957692 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.957982 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.957984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.957985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.958247 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:09.958410 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:09.958565 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:01:09.958731 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":93,"threads_started":1}
W20260810 23:01:09.960022 15834 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40194
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:01:09 dist-test-slave-6bbx krb5kdc[15596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:01:09.988392 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:09.979437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.15.56.1:45909)
0810 23:01:09.979589 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:09.979593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:09.979652 (+ 59us) server_negotiation.cc:407] Connection header received
0810 23:01:09.979891 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:09.979894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:09.979947 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:09.980035 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:09.980889 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.981507 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.982475 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:09.982725 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:09.982810 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:09.985488 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:09.985492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:09.985494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:09.985527 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:09.986918 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.987339 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.987342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.987343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.987426 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:09.987624 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:09.987626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:09.987627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:09.987835 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:09.987938 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:09.988126 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 23:01:09.988260 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:01:09.989450 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 15774
I20260810 23:01:10.004067 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 15978
I20260810 23:01:10.016847 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 16193
I20260810 23:01:10.031232 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 15617
2026-08-10T23:01:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2394 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:10 dist-test-slave-6bbx krb5kdc[16420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:10 dist-test-slave-6bbx krb5kdc[16420](info): set up 2 sockets
Aug 10 23:01:10 dist-test-slave-6bbx krb5kdc[16420](info): commencing operation
krb5kdc: starting...
W20260810 23:01:11.191860 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:11 dist-test-slave-6bbx krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:11.330961 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43153
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:43153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:11.446237 16440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:11.446542 16440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:11.446596 16440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:11.450634 16440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:11.450703 16440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:11.450730 16440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:11.450752 16440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:11.450771 16440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:11.456151 16440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:43153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:11.457284 16440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:11.458361 16440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:11.467464 16457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:11.467473 16451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:11.467464 16452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:11.467857 16440 server_base.cc:1034] running on GCE node
I20260810 23:01:11.468592 16440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:11.469808 16440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:11.470999 16440 hybrid_clock.cc:648] HybridClock initialized: now 1786402871470961 us; error 51 us; skew 500 ppm
Aug 10 23:01:11 dist-test-slave-6bbx krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402871, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:11.473958 16440 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:11.474931 16440 webserver.cc:460] Webserver started at http://127.15.56.62:35769/ using document root <none> and password file <none>
I20260810 23:01:11.475512 16440 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:11.475579 16440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:11.475813 16440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:11.477982 16440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "ed8e2da0ce1f44d1806926f52e72fb27"
format_stamp: "Formatted at 2026-08-10 23:01:11 on dist-test-slave-6bbx"
server_key: "be1eef71f5d87c95f083ed873ed76b7c"
server_key_iv: "a8ecf17da1c1737107e8277d37c505b8"
server_key_version: "encryptionkey@0"
I20260810 23:01:11.478559 16440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "ed8e2da0ce1f44d1806926f52e72fb27"
format_stamp: "Formatted at 2026-08-10 23:01:11 on dist-test-slave-6bbx"
server_key: "be1eef71f5d87c95f083ed873ed76b7c"
server_key_iv: "a8ecf17da1c1737107e8277d37c505b8"
server_key_version: "encryptionkey@0"
I20260810 23:01:11.482560 16440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 23:01:11.485333 16468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:11.486681 16440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:01:11.486804 16440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "ed8e2da0ce1f44d1806926f52e72fb27"
format_stamp: "Formatted at 2026-08-10 23:01:11 on dist-test-slave-6bbx"
server_key: "be1eef71f5d87c95f083ed873ed76b7c"
server_key_iv: "a8ecf17da1c1737107e8277d37c505b8"
server_key_version: "encryptionkey@0"
I20260810 23:01:11.486928 16440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:11.522261 16440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:11.525714 16440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:11.525905 16440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:11.533335 16440 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:43153
I20260810 23:01:11.533380 16551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:43153 every 8 connection(s)
I20260810 23:01:11.534322 16440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:11.537391 16552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:11.539064 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 16440
I20260810 23:01:11.539181 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:11.539465 15584 external_mini_cluster.cc:1428] Setting key 9434c55bdff256bfdaa9c7ad14fd4156
I20260810 23:01:11.544462 16552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27: Bootstrap starting.
I20260810 23:01:11.547062 16552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:11.547986 16552 log.cc:824] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:01:11 dist-test-slave-6bbx krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:11.550302 16552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27: No bootstrap required, opened a new log
I20260810 23:01:11.554476 16552 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8e2da0ce1f44d1806926f52e72fb27" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43153 } }
I20260810 23:01:11.554659 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:11.541074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.15.56.62:43153)
0810 23:01:11.541556 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:11.541572 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:11.541821 (+ 249us) server_negotiation.cc:407] Connection header received
0810 23:01:11.542563 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:11.542583 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:11.542968 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:11.543264 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:11.544172 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:11.545502 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:11.546682 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:11.547171 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:11.547334 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:11.550047 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:11.550053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:11.550068 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:11.550097 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:11.551797 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:11.552289 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:11.552294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:11.552299 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:11.552424 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:11.552704 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:11.552706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:11.552707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:11.553092 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:11.553289 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:11.553530 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 23:01:11.553843 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":150,"threads_started":1}
I20260810 23:01:11.554700 16552 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:11.554747 16552 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8e2da0ce1f44d1806926f52e72fb27, State: Initialized, Role: FOLLOWER
I20260810 23:01:11.555205 16552 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8e2da0ce1f44d1806926f52e72fb27" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43153 } }
I20260810 23:01:11.555343 16552 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:11.555411 16552 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:11.555511 16552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:11.556465 16552 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8e2da0ce1f44d1806926f52e72fb27" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43153 } }
I20260810 23:01:11.556806 16552 leader_election.cc:302] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed8e2da0ce1f44d1806926f52e72fb27; no voters:
I20260810 23:01:11.557112 16552 leader_election.cc:288] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:11.557196 16558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:11.557390 16558 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [term 1 LEADER]: Becoming Leader. State: Replica: ed8e2da0ce1f44d1806926f52e72fb27, State: Running, Role: LEADER
I20260810 23:01:11.557686 16558 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8e2da0ce1f44d1806926f52e72fb27" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43153 } }
I20260810 23:01:11.558431 16552 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:11.559605 16561 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed8e2da0ce1f44d1806926f52e72fb27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8e2da0ce1f44d1806926f52e72fb27" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43153 } } }
I20260810 23:01:11.559623 16562 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed8e2da0ce1f44d1806926f52e72fb27. Latest consensus state: current_term: 1 leader_uuid: "ed8e2da0ce1f44d1806926f52e72fb27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8e2da0ce1f44d1806926f52e72fb27" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43153 } } }
I20260810 23:01:11.559839 16562 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:11.560429 16561 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:11.560843 16573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:11.563768 16573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:11.569593 16573 catalog_manager.cc:1347] Generated new cluster ID: 3296a99d255c467f91239d1d1c6a0710
I20260810 23:01:11.569664 16573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:11.580488 16573 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:11.581804 16573 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:11.592151 16573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed8e2da0ce1f44d1806926f52e72fb27: Generated new TSK 0
I20260810 23:01:11.593094 16573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:01:11.622421 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43153
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:11.741209 16591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:11.741518 16591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:11.741566 16591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:11.745730 16591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:11.745803 16591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:11.745867 16591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:11.751372 16591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:11.752529 16591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:11.753587 16591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:11.764009 16602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:11.764009 16596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:11.764024 16598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:11.764408 16591 server_base.cc:1034] running on GCE node
I20260810 23:01:11.764849 16591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:11.765523 16591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:11.766683 16591 hybrid_clock.cc:648] HybridClock initialized: now 1786402871766664 us; error 28 us; skew 500 ppm
Aug 10 23:01:11 dist-test-slave-6bbx krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402871, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:11.769546 16591 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:01:11.770462 16591 webserver.cc:460] Webserver started at http://127.15.56.1:38477/ using document root <none> and password file <none>
I20260810 23:01:11.771039 16591 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:11.771106 16591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:11.771325 16591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:11.773479 16591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "dd61ca084209449eac756f74cef412ba"
format_stamp: "Formatted at 2026-08-10 23:01:11 on dist-test-slave-6bbx"
server_key: "dac855df028751d597a4295c29d9313f"
server_key_iv: "905b20c7d3b5732096c411110b72ec9d"
server_key_version: "encryptionkey@0"
I20260810 23:01:11.774042 16591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "dd61ca084209449eac756f74cef412ba"
format_stamp: "Formatted at 2026-08-10 23:01:11 on dist-test-slave-6bbx"
server_key: "dac855df028751d597a4295c29d9313f"
server_key_iv: "905b20c7d3b5732096c411110b72ec9d"
server_key_version: "encryptionkey@0"
I20260810 23:01:11.778012 16591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:01:11.780808 16617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:11.782271 16591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:11.782433 16591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "dd61ca084209449eac756f74cef412ba"
format_stamp: "Formatted at 2026-08-10 23:01:11 on dist-test-slave-6bbx"
server_key: "dac855df028751d597a4295c29d9313f"
server_key_iv: "905b20c7d3b5732096c411110b72ec9d"
server_key_version: "encryptionkey@0"
I20260810 23:01:11.782553 16591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:11.792683 16591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:11.795703 16591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:11.795887 16591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:11.796499 16591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:11.797343 16591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:11.797394 16591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:11.797446 16591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:11.797474 16591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:11.806291 16591 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:42723
I20260810 23:01:11.806336 16797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:42723 every 8 connection(s)
I20260810 23:01:11.807220 16591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:11.809689 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 16591
I20260810 23:01:11.809792 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:11.810098 15584 external_mini_cluster.cc:1428] Setting key f0e27ff528ad7bffbd8e037603f31b15
Aug 10 23:01:11 dist-test-slave-6bbx krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402871, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:01:11.822800 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:11.809176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:40421 (local address 127.15.56.62:43153)
0810 23:01:11.809329 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:11.809334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:11.810351 (+ 1017us) server_negotiation.cc:407] Connection header received
0810 23:01:11.811846 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:11.811849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:11.811893 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:11.811951 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:11.813489 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:11.814284 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:11.815500 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:11.815844 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:11.815944 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:11.818423 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:11.818427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:11.818430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:11.818461 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:11.820167 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:11.820799 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:11.820804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:11.820806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:11.820908 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:11.821377 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:11.821380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:11.821381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:11.821603 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:11.821723 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:11.822519 (+ 796us) server_negotiation.cc:299] Negotiation successful
0810 23:01:11.822657 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:01:11.823892 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:11.809624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43153 (local address 127.15.56.1:40421)
0810 23:01:11.810164 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:11.810236 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:11.811609 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:11.812138 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:11.812147 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:11.812637 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:11.813308 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:11.813327 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:11.814427 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:11.814430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:11.815371 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:11.815382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:11.816011 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:11.816014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:11.816160 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:11.816750 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:11.816768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:11.818232 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:11.820312 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:11.820318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:11.820334 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:11.820632 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:11.821039 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:11.821042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:11.821044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:11.821246 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:11.821905 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:11.821911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:11.822212 (+ 301us) client_negotiation.cc:769] Sending connection context
0810 23:01:11.822435 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 23:01:11.822841 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
I20260810 23:01:11.825366 16799 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43153
I20260810 23:01:11.825651 16799 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:11.826215 16799 heartbeater.cc:511] Master 127.15.56.62:43153 requested a full tablet report, sending...
I20260810 23:01:11.827939 16493 ts_manager.cc:194] Registered new tserver with Master: dd61ca084209449eac756f74cef412ba (127.15.56.1:42723)
I20260810 23:01:11.829154 16493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:40421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:01:11.843186 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43153
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:11.962741 16805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:11.963029 16805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:11.963075 16805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:11.967501 16805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:11.967564 16805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:11.967625 16805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:11.973114 16805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:11.974215 16805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:11.975242 16805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:11.985462 16810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:11.985556 16811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:11.985462 16813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:11.985777 16805 server_base.cc:1034] running on GCE node
I20260810 23:01:11.986328 16805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:11.987061 16805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:11.988224 16805 hybrid_clock.cc:648] HybridClock initialized: now 1786402871988190 us; error 43 us; skew 500 ppm
Aug 10 23:01:11 dist-test-slave-6bbx krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402871, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:11.991063 16805 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:01:11.992241 16805 webserver.cc:460] Webserver started at http://127.15.56.2:45659/ using document root <none> and password file <none>
I20260810 23:01:11.992839 16805 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:11.992910 16805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:11.993127 16805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:11.995280 16805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "cd6e03e84abb4f809ade419012c8dee2"
format_stamp: "Formatted at 2026-08-10 23:01:11 on dist-test-slave-6bbx"
server_key: "947bd29566675b6b5de317b1535f1c3a"
server_key_iv: "0eac14b5336d190153a65dc73fbef317"
server_key_version: "encryptionkey@0"
I20260810 23:01:11.995905 16805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "cd6e03e84abb4f809ade419012c8dee2"
format_stamp: "Formatted at 2026-08-10 23:01:11 on dist-test-slave-6bbx"
server_key: "947bd29566675b6b5de317b1535f1c3a"
server_key_iv: "0eac14b5336d190153a65dc73fbef317"
server_key_version: "encryptionkey@0"
I20260810 23:01:11.999642 16805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:01:12.002151 16824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:12.003398 16805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:12.003516 16805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "cd6e03e84abb4f809ade419012c8dee2"
format_stamp: "Formatted at 2026-08-10 23:01:11 on dist-test-slave-6bbx"
server_key: "947bd29566675b6b5de317b1535f1c3a"
server_key_iv: "0eac14b5336d190153a65dc73fbef317"
server_key_version: "encryptionkey@0"
I20260810 23:01:12.003618 16805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:12.030499 16805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:12.034138 16805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:12.034359 16805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:12.035037 16805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:12.035986 16805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:12.036039 16805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:12.036094 16805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:12.036124 16805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:12.045267 16805 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:36799
I20260810 23:01:12.045347 16994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:36799 every 8 connection(s)
I20260810 23:01:12.046258 16805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:12.051270 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 16805
I20260810 23:01:12.051391 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:12.051647 15584 external_mini_cluster.cc:1428] Setting key be51f8bf4c4d714177c93d9b79753610
Aug 10 23:01:12 dist-test-slave-6bbx krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402871, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:12.062213 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.048282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:36109 (local address 127.15.56.62:43153)
0810 23:01:12.048418 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:12.048421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:12.049404 (+ 983us) server_negotiation.cc:407] Connection header received
0810 23:01:12.050836 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:12.050840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:12.050908 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:12.050984 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:12.052529 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.053165 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.054650 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.054975 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.055095 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.057984 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:12.057990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:12.057993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:12.058026 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:12.059881 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.060481 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.060486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.060488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.060587 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.060972 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.060975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.060977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.061243 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:12.061350 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:12.061874 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 23:01:12.062015 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:01:12.063060 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.048696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43153 (local address 127.15.56.2:36109)
0810 23:01:12.049231 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:12.049284 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:12.050558 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:12.051203 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:12.051211 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:12.051657 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:12.052361 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.052384 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.053292 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.053296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.054511 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.054522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.055246 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.055250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.055423 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.056056 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:12.056074 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:12.057781 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:12.060017 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:12.060022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:12.060037 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:12.060373 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:12.060697 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:12.060700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:12.060702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:12.060874 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:12.061354 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:12.061360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:12.061623 (+ 263us) client_negotiation.cc:769] Sending connection context
0810 23:01:12.061796 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 23:01:12.062081 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":143,"threads_started":1}
I20260810 23:01:12.064522 16995 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43153
I20260810 23:01:12.064790 16995 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:12.065351 16995 heartbeater.cc:511] Master 127.15.56.62:43153 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
I20260810 23:01:12.066565 16493 ts_manager.cc:194] Registered new tserver with Master: cd6e03e84abb4f809ade419012c8dee2 (127.15.56.2:36799)
I20260810 23:01:12.067309 16493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:36109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:01:12.083314 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43153
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:12.205284 17002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:12.205597 17002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:12.205672 17002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:12.209945 17002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:12.210023 17002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:12.210111 17002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:12.215799 17002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43153
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:12.217012 17002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:12.218143 17002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:12.228461 17009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:12.228495 17008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:12.228487 17011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:12.228871 17002 server_base.cc:1034] running on GCE node
I20260810 23:01:12.229439 17002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:12.230157 17002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:12.231348 17002 hybrid_clock.cc:648] HybridClock initialized: now 1786402872231313 us; error 48 us; skew 500 ppm
Aug 10 23:01:12 dist-test-slave-6bbx krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402872, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:12.234364 17002 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:01:12.235324 17002 webserver.cc:460] Webserver started at http://127.15.56.3:38481/ using document root <none> and password file <none>
I20260810 23:01:12.235965 17002 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:12.236038 17002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:12.236274 17002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:12.238511 17002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "9f937ae2be7e46f3ace3286198d17958"
format_stamp: "Formatted at 2026-08-10 23:01:12 on dist-test-slave-6bbx"
server_key: "43ebf79d9b7bdade4c839a957c25484c"
server_key_iv: "3202d702f12ed2f5ea1e7c4d2df5f9b1"
server_key_version: "encryptionkey@0"
I20260810 23:01:12.239223 17002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "9f937ae2be7e46f3ace3286198d17958"
format_stamp: "Formatted at 2026-08-10 23:01:12 on dist-test-slave-6bbx"
server_key: "43ebf79d9b7bdade4c839a957c25484c"
server_key_iv: "3202d702f12ed2f5ea1e7c4d2df5f9b1"
server_key_version: "encryptionkey@0"
I20260810 23:01:12.243453 17002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:01:12.246232 17023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:12.247607 17002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:12.247761 17002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "9f937ae2be7e46f3ace3286198d17958"
format_stamp: "Formatted at 2026-08-10 23:01:12 on dist-test-slave-6bbx"
server_key: "43ebf79d9b7bdade4c839a957c25484c"
server_key_iv: "3202d702f12ed2f5ea1e7c4d2df5f9b1"
server_key_version: "encryptionkey@0"
I20260810 23:01:12.247885 17002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:12.260226 17002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:12.263530 17002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:12.263751 17002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:12.264441 17002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:12.265357 17002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:12.265405 17002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:12.265450 17002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:12.265478 17002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:12.274176 17002 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:39801
I20260810 23:01:12.274189 17213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:39801 every 8 connection(s)
I20260810 23:01:12.275133 17002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:12.281548 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 17002
I20260810 23:01:12.281673 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:12.281950 15584 external_mini_cluster.cc:1428] Setting key 69c1ddb7b151f0f466a9b0bf560f6266
Aug 10 23:01:12 dist-test-slave-6bbx krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402872, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:12.290113 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.277074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:41817 (local address 127.15.56.62:43153)
0810 23:01:12.277228 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:12.277231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:12.278203 (+ 972us) server_negotiation.cc:407] Connection header received
0810 23:01:12.279560 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:12.279564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:12.279615 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:12.279758 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:12.281213 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.281790 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.283467 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.283790 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.283890 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.286668 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:12.286671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:12.286673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:12.286702 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:12.288028 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.288536 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.288540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.288541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.288619 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.288967 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.288970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.288972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.289163 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:12.289243 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:12.289808 (+ 565us) server_negotiation.cc:299] Negotiation successful
0810 23:01:12.289945 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:01:12.291033 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.277484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43153 (local address 127.15.56.3:41817)
0810 23:01:12.278028 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:12.278076 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:12.279327 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:12.279921 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:12.279929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:12.280466 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:12.281025 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.281042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.281947 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.281951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.283305 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.283318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.283977 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.283981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.284161 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.284796 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:12.284815 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:12.286490 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:12.288141 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:12.288148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:12.288167 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:12.288436 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:12.288712 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:12.288716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:12.288718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:12.288878 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:12.289276 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:12.289283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:12.289563 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 23:01:12.289748 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 23:01:12.290125 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":129,"threads_started":1}
I20260810 23:01:12.292276 17215 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43153
I20260810 23:01:12.292557 17215 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:12.293143 17215 heartbeater.cc:511] Master 127.15.56.62:43153 requested a full tablet report, sending...
I20260810 23:01:12.294361 16493 ts_manager.cc:194] Registered new tserver with Master: 9f937ae2be7e46f3ace3286198d17958 (127.15.56.3:39801)
I20260810 23:01:12.295082 16493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:41817
I20260810 23:01:12.296505 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:01:12.308224 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.300249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.15.56.62:43153)
0810 23:01:12.300396 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:12.300401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:12.300575 (+ 174us) server_negotiation.cc:407] Connection header received
0810 23:01:12.300703 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:12.300706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:12.300753 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:12.300842 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:12.301586 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.302131 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.302976 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.303197 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.303286 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.304874 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:12.304878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:12.304881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:12.304912 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:12.306316 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.306845 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.306849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.306851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.306939 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.307214 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.307217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.307219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.307477 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:12.307604 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:12.307923 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 23:01:12.308047 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:01:12.313861 16492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:01:12.317060 16492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:12.332957 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.326423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:39801 (local address 127.0.0.1:52296)
0810 23:01:12.327309 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:12.327366 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:12.327494 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:12.328323 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:12.328328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:12.328348 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:12.328455 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.328459 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.329776 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.329779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.331379 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.331386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.332309 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.332313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.332440 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.332483 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 23:01:12.332611 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:01:12.332758 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":64,"thread_start_us":90,"threads_started":1}
I20260810 23:01:12.333052 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.327644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.15.56.2:36799)
0810 23:01:12.328090 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:12.328112 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:12.328137 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:01:12.328219 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:12.328223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:12.328413 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:12.328567 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:12.329073 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.330092 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.331426 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.332321 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.332430 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.332575 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:12.332768 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:01:12.332883 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":104,"threads_started":1}
I20260810 23:01:12.333272 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.326769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.15.56.1:42723)
0810 23:01:12.327604 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:12.327611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:12.328173 (+ 562us) server_negotiation.cc:407] Connection header received
0810 23:01:12.328250 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:12.328255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:12.328413 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:12.328567 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:12.328792 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.329623 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.331484 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.332321 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.332430 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.332575 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:12.333028 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 23:01:12.333130 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":108,"threads_started":1}
I20260810 23:01:12.333508 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.327241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:36799 (local address 127.0.0.1:51848)
0810 23:01:12.327863 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:12.327876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:12.327972 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:12.328756 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:12.328759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:12.328779 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:12.328893 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.328899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.330371 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.330375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.331297 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.331307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.332532 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.332535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.332630 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.332651 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:01:12.332716 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:01:12.332774 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":8,"thread_start_us":81,"threads_started":1}
I20260810 23:01:12.333555 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.326554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:42723 (local address 127.0.0.1:60726)
0810 23:01:12.328050 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:12.328064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:12.328155 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:12.328551 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:12.328554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:12.328571 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:12.328654 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.328660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.329750 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.329754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.331346 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.331359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.332641 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.332644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.332732 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.332757 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:01:12.332826 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 23:01:12.332895 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"mutex_wait_us":160,"thread_start_us":118,"threads_started":1}
I20260810 23:01:12.334568 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.326940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.15.56.3:39801)
0810 23:01:12.327401 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:12.327413 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:12.327444 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:01:12.327733 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:12.327740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:12.327966 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:12.328126 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:12.328570 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.329648 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.331505 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.332175 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.332281 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.332453 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:12.332864 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 23:01:12.332970 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":124,"threads_started":1}
I20260810 23:01:12.336966 17099 tablet_service.cc:1467] Processing CreateTablet for tablet 6e465305f7dd421a9355a9e4b4c9397f (DEFAULT_TABLE table=test-table [id=17476ae7c56c4f3290e179fa92eabb1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:12.336938 16692 tablet_service.cc:1467] Processing CreateTablet for tablet 6e465305f7dd421a9355a9e4b4c9397f (DEFAULT_TABLE table=test-table [id=17476ae7c56c4f3290e179fa92eabb1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:12.336941 16902 tablet_service.cc:1467] Processing CreateTablet for tablet 6e465305f7dd421a9355a9e4b4c9397f (DEFAULT_TABLE table=test-table [id=17476ae7c56c4f3290e179fa92eabb1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:12.338090 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e465305f7dd421a9355a9e4b4c9397f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:12.338089 17099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e465305f7dd421a9355a9e4b4c9397f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:12.338090 16902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e465305f7dd421a9355a9e4b4c9397f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:12.344830 17256 tablet_bootstrap.cc:492] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958: Bootstrap starting.
I20260810 23:01:12.346483 17258 tablet_bootstrap.cc:492] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba: Bootstrap starting.
I20260810 23:01:12.347230 17256 tablet_bootstrap.cc:654] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:12.348059 17262 tablet_bootstrap.cc:492] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2: Bootstrap starting.
I20260810 23:01:12.348538 17256 log.cc:824] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:12.348750 17258 tablet_bootstrap.cc:654] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:12.349651 17258 log.cc:824] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:12.350812 17262 tablet_bootstrap.cc:654] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:12.351038 17256 tablet_bootstrap.cc:492] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958: No bootstrap required, opened a new log
I20260810 23:01:12.351243 17256 ts_tablet_manager.cc:1397] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 23:01:12.351708 17262 log.cc:824] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:12.351933 17258 tablet_bootstrap.cc:492] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba: No bootstrap required, opened a new log
I20260810 23:01:12.352151 17258 ts_tablet_manager.cc:1397] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:01:12.353865 17262 tablet_bootstrap.cc:492] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2: No bootstrap required, opened a new log
I20260810 23:01:12.354082 17262 ts_tablet_manager.cc:1397] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:01:12.354696 17256 raft_consensus.cc:348] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f937ae2be7e46f3ace3286198d17958" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 39801 } } peers { permanent_uuid: "dd61ca084209449eac756f74cef412ba" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 42723 } } peers { permanent_uuid: "cd6e03e84abb4f809ade419012c8dee2" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36799 } }
I20260810 23:01:12.354949 17256 raft_consensus.cc:374] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:12.355005 17256 raft_consensus.cc:729] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f937ae2be7e46f3ace3286198d17958, State: Initialized, Role: FOLLOWER
I20260810 23:01:12.355469 17256 consensus_queue.cc:260] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f937ae2be7e46f3ace3286198d17958" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 39801 } } peers { permanent_uuid: "dd61ca084209449eac756f74cef412ba" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 42723 } } peers { permanent_uuid: "cd6e03e84abb4f809ade419012c8dee2" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36799 } }
I20260810 23:01:12.356912 17256 ts_tablet_manager.cc:1428] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:01:12.356813 17258 raft_consensus.cc:348] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f937ae2be7e46f3ace3286198d17958" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 39801 } } peers { permanent_uuid: "dd61ca084209449eac756f74cef412ba" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 42723 } } peers { permanent_uuid: "cd6e03e84abb4f809ade419012c8dee2" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36799 } }
I20260810 23:01:12.357107 17258 raft_consensus.cc:374] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:12.357168 17258 raft_consensus.cc:729] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd61ca084209449eac756f74cef412ba, State: Initialized, Role: FOLLOWER
I20260810 23:01:12.357306 17215 heartbeater.cc:503] Master 127.15.56.62:43153 was elected leader, sending a full tablet report...
I20260810 23:01:12.357681 17258 consensus_queue.cc:260] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f937ae2be7e46f3ace3286198d17958" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 39801 } } peers { permanent_uuid: "dd61ca084209449eac756f74cef412ba" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 42723 } } peers { permanent_uuid: "cd6e03e84abb4f809ade419012c8dee2" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36799 } }
I20260810 23:01:12.357957 17262 raft_consensus.cc:348] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f937ae2be7e46f3ace3286198d17958" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 39801 } } peers { permanent_uuid: "dd61ca084209449eac756f74cef412ba" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 42723 } } peers { permanent_uuid: "cd6e03e84abb4f809ade419012c8dee2" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36799 } }
I20260810 23:01:12.358289 17262 raft_consensus.cc:374] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:12.358352 17262 raft_consensus.cc:729] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6e03e84abb4f809ade419012c8dee2, State: Initialized, Role: FOLLOWER
I20260810 23:01:12.358402 16799 heartbeater.cc:503] Master 127.15.56.62:43153 was elected leader, sending a full tablet report...
I20260810 23:01:12.358863 17262 consensus_queue.cc:260] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f937ae2be7e46f3ace3286198d17958" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 39801 } } peers { permanent_uuid: "dd61ca084209449eac756f74cef412ba" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 42723 } } peers { permanent_uuid: "cd6e03e84abb4f809ade419012c8dee2" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36799 } }
I20260810 23:01:12.359596 16995 heartbeater.cc:503] Master 127.15.56.62:43153 was elected leader, sending a full tablet report...
I20260810 23:01:12.359999 17262 ts_tablet_manager.cc:1428] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:01:12.360805 17258 ts_tablet_manager.cc:1428] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260810 23:01:12.367765 17266 raft_consensus.cc:482] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:12.367969 17266 raft_consensus.cc:504] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f937ae2be7e46f3ace3286198d17958" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 39801 } } peers { permanent_uuid: "dd61ca084209449eac756f74cef412ba" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 42723 } } peers { permanent_uuid: "cd6e03e84abb4f809ade419012c8dee2" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36799 } }
I20260810 23:01:12.369125 17266 leader_election.cc:288] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd61ca084209449eac756f74cef412ba (127.15.56.1:42723), cd6e03e84abb4f809ade419012c8dee2 (127.15.56.2:36799)
I20260810 23:01:12.372642 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.369505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:37389 (local address 127.15.56.1:42723)
0810 23:01:12.369648 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:12.369653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:12.369671 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:01:12.369733 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:12.369736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:12.369783 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:12.369858 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:12.370198 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.370694 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.371718 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.372129 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.372228 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.372292 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:12.372457 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 23:01:12.372530 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:01:12.372642 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.369364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:42723 (local address 127.15.56.3:37389)
0810 23:01:12.369511 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:12.369528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:12.369627 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:12.369972 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:12.369975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:12.369991 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:12.370085 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.370090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.370800 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.370803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.371568 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.371576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.372252 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.372255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.372352 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.372370 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 23:01:12.372433 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:01:12.372504 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 23:01:12.373088 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.369601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:36799 (local address 127.15.56.3:45411)
0810 23:01:12.370006 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:12.370017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:12.370111 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:12.370319 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:12.370321 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:12.370336 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:12.370455 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.370459 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.371137 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.371140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.372029 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:12.372039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.372688 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.372692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:12.372793 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.372817 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:01:12.372887 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:01:12.372946 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":177,"threads_started":1}
I20260810 23:01:12.373231 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.369779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:45411 (local address 127.15.56.2:36799)
0810 23:01:12.369901 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:12.369906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:12.370089 (+ 183us) server_negotiation.cc:407] Connection header received
0810 23:01:12.370193 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:12.370196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:12.370238 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:12.370296 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:12.370549 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.371032 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.372153 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.372545 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.372637 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.372696 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:12.373043 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 23:01:12.373109 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:01:12.373329 16733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e465305f7dd421a9355a9e4b4c9397f" candidate_uuid: "9f937ae2be7e46f3ace3286198d17958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd61ca084209449eac756f74cef412ba" is_pre_election: true
I20260810 23:01:12.373690 16733 raft_consensus.cc:2454] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f937ae2be7e46f3ace3286198d17958 in term 0.
I20260810 23:01:12.373775 16931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e465305f7dd421a9355a9e4b4c9397f" candidate_uuid: "9f937ae2be7e46f3ace3286198d17958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6e03e84abb4f809ade419012c8dee2" is_pre_election: true
I20260810 23:01:12.374059 16931 raft_consensus.cc:2454] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f937ae2be7e46f3ace3286198d17958 in term 0.
I20260810 23:01:12.374220 17034 leader_election.cc:302] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f937ae2be7e46f3ace3286198d17958, dd61ca084209449eac756f74cef412ba; no voters:
I20260810 23:01:12.374516 17266 raft_consensus.cc:2781] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:12.374599 17266 raft_consensus.cc:482] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:12.374673 17266 raft_consensus.cc:3037] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:12.375976 17266 raft_consensus.cc:504] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f937ae2be7e46f3ace3286198d17958" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 39801 } } peers { permanent_uuid: "dd61ca084209449eac756f74cef412ba" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 42723 } } peers { permanent_uuid: "cd6e03e84abb4f809ade419012c8dee2" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36799 } }
I20260810 23:01:12.376278 17266 leader_election.cc:288] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [CANDIDATE]: Term 1 election: Requested vote from peers dd61ca084209449eac756f74cef412ba (127.15.56.1:42723), cd6e03e84abb4f809ade419012c8dee2 (127.15.56.2:36799)
I20260810 23:01:12.376624 16733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e465305f7dd421a9355a9e4b4c9397f" candidate_uuid: "9f937ae2be7e46f3ace3286198d17958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd61ca084209449eac756f74cef412ba"
I20260810 23:01:12.376713 16931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e465305f7dd421a9355a9e4b4c9397f" candidate_uuid: "9f937ae2be7e46f3ace3286198d17958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6e03e84abb4f809ade419012c8dee2"
I20260810 23:01:12.376749 16733 raft_consensus.cc:3037] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:12.376847 16931 raft_consensus.cc:3037] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:12.377825 16733 raft_consensus.cc:2454] T 6e465305f7dd421a9355a9e4b4c9397f P dd61ca084209449eac756f74cef412ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f937ae2be7e46f3ace3286198d17958 in term 1.
I20260810 23:01:12.377925 16931 raft_consensus.cc:2454] T 6e465305f7dd421a9355a9e4b4c9397f P cd6e03e84abb4f809ade419012c8dee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f937ae2be7e46f3ace3286198d17958 in term 1.
I20260810 23:01:12.378110 17034 leader_election.cc:302] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f937ae2be7e46f3ace3286198d17958, dd61ca084209449eac756f74cef412ba; no voters:
I20260810 23:01:12.378278 17266 raft_consensus.cc:2781] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:12.378500 17266 raft_consensus.cc:686] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [term 1 LEADER]: Becoming Leader. State: Replica: 9f937ae2be7e46f3ace3286198d17958, State: Running, Role: LEADER
I20260810 23:01:12.378834 17266 consensus_queue.cc:237] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f937ae2be7e46f3ace3286198d17958" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 39801 } } peers { permanent_uuid: "dd61ca084209449eac756f74cef412ba" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 42723 } } peers { permanent_uuid: "cd6e03e84abb4f809ade419012c8dee2" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36799 } }
I20260810 23:01:12.382208 16492 catalog_manager.cc:5495] T 6e465305f7dd421a9355a9e4b4c9397f P 9f937ae2be7e46f3ace3286198d17958 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f937ae2be7e46f3ace3286198d17958 (127.15.56.3). New cstate: current_term: 1 leader_uuid: "9f937ae2be7e46f3ace3286198d17958" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f937ae2be7e46f3ace3286198d17958" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 39801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd61ca084209449eac756f74cef412ba" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd6e03e84abb4f809ade419012c8dee2" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36799 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:01:12 dist-test-slave-6bbx krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:01:12.399195 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.389811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.15.56.1:42723)
0810 23:01:12.389964 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:12.389969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:12.390099 (+ 130us) server_negotiation.cc:407] Connection header received
0810 23:01:12.390282 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:12.390286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:12.390335 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:12.390437 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:12.391115 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.391746 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.392571 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.392828 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.392918 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.395408 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:12.395425 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:12.395443 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:12.395471 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:12.397386 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.397824 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.397830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.397832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.397935 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.398228 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.398232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.398234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.398510 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:12.398746 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:12.398908 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 23:01:12.399030 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 23:01:12 dist-test-slave-6bbx krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:01:12 dist-test-slave-6bbx krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:01:12.427806 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.419267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.15.56.1:42723)
0810 23:01:12.419441 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:12.419446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:12.419570 (+ 124us) server_negotiation.cc:407] Connection header received
0810 23:01:12.419762 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:12.419765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:12.419816 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:12.419888 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:12.420605 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.421120 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.421942 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.422175 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.422273 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.424693 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:12.424696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:12.424698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:12.424735 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:12.425970 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.426419 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.426423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.426424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.426521 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.426837 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.426840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.426842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.427032 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:12.427146 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:12.427506 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 23:01:12.427621 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 23:01:12 dist-test-slave-6bbx krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:01:12 dist-test-slave-6bbx krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:01:12.454043 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:12.446105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.15.56.1:42723)
0810 23:01:12.446261 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:12.446266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:12.446418 (+ 152us) server_negotiation.cc:407] Connection header received
0810 23:01:12.446568 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:12.446571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:12.446614 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:12.446706 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:12.447441 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.448036 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.448893 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:12.449102 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:12.449180 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:12.451496 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:12.451500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:12.451502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:12.451528 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:12.452795 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.453146 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.453148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.453149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.453215 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:12.453405 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:12.453407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:12.453408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:12.453561 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:12.453652 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:12.453819 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 23:01:12.453915 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 23:01:12.455528 16659 tablet_service.cc:2908] Rejecting scan request for tablet 6e465305f7dd421a9355a9e4b4c9397f: Uninitialized: clean time has not yet been initialized
I20260810 23:01:12.457991 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 16591
I20260810 23:01:12.476853 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 16805
I20260810 23:01:12.488866 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 17002
I20260810 23:01:12.511437 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 16440
2026-08-10T23:01:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2483 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:12 dist-test-slave-6bbx krb5kdc[17299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:12 dist-test-slave-6bbx krb5kdc[17299](info): set up 2 sockets
Aug 10 23:01:12 dist-test-slave-6bbx krb5kdc[17299](info): commencing operation
krb5kdc: starting...
W20260810 23:01:13.706039 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:13 dist-test-slave-6bbx krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:13.839399 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:39895
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:13.955649 17322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:13.955955 17322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:13.956010 17322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:13.959967 17322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:13.960022 17322 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:01:13.960045 17322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:13.960068 17322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:13.960104 17322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:13.960132 17322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:13.965412 17322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:39895
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:39895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:13.966519 17322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:13.967578 17322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:13.976223 17332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:13.976212 17333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:13.976212 17335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:13.976274 17322 server_base.cc:1034] running on GCE node
I20260810 23:01:13.977035 17322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:13.978143 17322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:13.979314 17322 hybrid_clock.cc:648] HybridClock initialized: now 1786402873979298 us; error 31 us; skew 500 ppm
Aug 10 23:01:13 dist-test-slave-6bbx krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402873, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:13.982141 17322 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:13.983094 17322 webserver.cc:460] Webserver started at http://127.15.56.62:33611/ using document root <none> and password file <none>
I20260810 23:01:13.983703 17322 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:13.983779 17322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:13.983994 17322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:13.986106 17322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "4579d60920e845b28fd5bffc47bbf2af"
format_stamp: "Formatted at 2026-08-10 23:01:13 on dist-test-slave-6bbx"
server_key: "6f997d07191b11651598616674daf39c"
server_key_iv: "ee8c37a8ac6a0bacab97e338b2be52b9"
server_key_version: "encryptionkey@0"
I20260810 23:01:13.986678 17322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "4579d60920e845b28fd5bffc47bbf2af"
format_stamp: "Formatted at 2026-08-10 23:01:13 on dist-test-slave-6bbx"
server_key: "6f997d07191b11651598616674daf39c"
server_key_iv: "ee8c37a8ac6a0bacab97e338b2be52b9"
server_key_version: "encryptionkey@0"
I20260810 23:01:13.990576 17322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 23:01:13.993072 17349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:13.994328 17322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:01:13.994450 17322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "4579d60920e845b28fd5bffc47bbf2af"
format_stamp: "Formatted at 2026-08-10 23:01:13 on dist-test-slave-6bbx"
server_key: "6f997d07191b11651598616674daf39c"
server_key_iv: "ee8c37a8ac6a0bacab97e338b2be52b9"
server_key_version: "encryptionkey@0"
I20260810 23:01:13.994546 17322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:14.011152 17322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:14.014076 17322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:14.014276 17322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:14.022603 17322 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:39895
I20260810 23:01:14.022620 17449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:39895 every 8 connection(s)
I20260810 23:01:14.023592 17322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:14.026467 17450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:14.027731 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 17322
I20260810 23:01:14.027848 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:14.028092 15584 external_mini_cluster.cc:1428] Setting key 45b3572d33313b4f3fb24b4c5ef0d9b6
I20260810 23:01:14.032838 17450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af: Bootstrap starting.
I20260810 23:01:14.035600 17450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:14.036407 17450 log.cc:824] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:01:14 dist-test-slave-6bbx krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:14.038640 17450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af: No bootstrap required, opened a new log
I20260810 23:01:14.041719 17450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4579d60920e845b28fd5bffc47bbf2af" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 39895 } }
I20260810 23:01:14.041960 17450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:14.042017 17450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4579d60920e845b28fd5bffc47bbf2af, State: Initialized, Role: FOLLOWER
I20260810 23:01:14.042596 17450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4579d60920e845b28fd5bffc47bbf2af" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 39895 } }
I20260810 23:01:14.042747 17450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:14.042817 17450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:14.042919 17450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:14.043283 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.029629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.15.56.62:39895)
0810 23:01:14.030052 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:14.030069 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:14.030108 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:01:14.030766 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:14.030785 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:14.031122 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:14.031413 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:14.032413 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.033492 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.034641 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.035124 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.035243 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.037949 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:14.037954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:14.037969 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:14.037997 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:14.040322 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.040758 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.040763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.040768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.040886 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.041157 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.041161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.041162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.041581 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:14.041791 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:14.042075 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 23:01:14.042418 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260810 23:01:14.043978 17450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4579d60920e845b28fd5bffc47bbf2af" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 39895 } }
I20260810 23:01:14.044312 17450 leader_election.cc:302] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4579d60920e845b28fd5bffc47bbf2af; no voters:
I20260810 23:01:14.044673 17450 leader_election.cc:288] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:14.044783 17460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:14.045003 17460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [term 1 LEADER]: Becoming Leader. State: Replica: 4579d60920e845b28fd5bffc47bbf2af, State: Running, Role: LEADER
I20260810 23:01:14.045326 17460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4579d60920e845b28fd5bffc47bbf2af" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 39895 } }
I20260810 23:01:14.045969 17450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:14.047156 17463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4579d60920e845b28fd5bffc47bbf2af. Latest consensus state: current_term: 1 leader_uuid: "4579d60920e845b28fd5bffc47bbf2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4579d60920e845b28fd5bffc47bbf2af" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 39895 } } }
I20260810 23:01:14.047298 17463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:14.047554 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4579d60920e845b28fd5bffc47bbf2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4579d60920e845b28fd5bffc47bbf2af" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 39895 } } }
I20260810 23:01:14.047648 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:14.047787 17473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:14.050940 17473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:14.057181 17473 catalog_manager.cc:1347] Generated new cluster ID: aae7ee524b654e56a3d1de30b561bef3
I20260810 23:01:14.057274 17473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:14.063625 17473 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:14.065122 17473 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:14.070698 17473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af: Generated new TSK 0
I20260810 23:01:14.071420 17473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:01:14.108121 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:39895
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:14.225268 17490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:14.225524 17490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:14.225584 17490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:14.229555 17490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:14.229617 17490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:14.229672 17490 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:01:14.229728 17490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:14.235033 17490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.56.62:39895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:14.236166 17490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:14.237238 17490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:14.247005 17499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:14.247001 17500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:14.247007 17503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:14.247543 17490 server_base.cc:1034] running on GCE node
I20260810 23:01:14.247977 17490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:14.248658 17490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:14.249823 17490 hybrid_clock.cc:648] HybridClock initialized: now 1786402874249804 us; error 31 us; skew 500 ppm
Aug 10 23:01:14 dist-test-slave-6bbx krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:14.252686 17490 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:01:14.253587 17490 webserver.cc:460] Webserver started at http://127.15.56.1:34535/ using document root <none> and password file <none>
I20260810 23:01:14.254117 17490 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:14.254186 17490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:14.254377 17490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:14.256505 17490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "bfa84e13a7c14495b3a496843569dd5e"
format_stamp: "Formatted at 2026-08-10 23:01:14 on dist-test-slave-6bbx"
server_key: "16bca8b6aa6f698d6d39dcfd6372f59f"
server_key_iv: "c7dc77b8e2eafccef704d3b40965fb36"
server_key_version: "encryptionkey@0"
I20260810 23:01:14.257026 17490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "bfa84e13a7c14495b3a496843569dd5e"
format_stamp: "Formatted at 2026-08-10 23:01:14 on dist-test-slave-6bbx"
server_key: "16bca8b6aa6f698d6d39dcfd6372f59f"
server_key_iv: "c7dc77b8e2eafccef704d3b40965fb36"
server_key_version: "encryptionkey@0"
I20260810 23:01:14.260762 17490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:01:14.263276 17514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:14.264613 17490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:01:14.264726 17490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "bfa84e13a7c14495b3a496843569dd5e"
format_stamp: "Formatted at 2026-08-10 23:01:14 on dist-test-slave-6bbx"
server_key: "16bca8b6aa6f698d6d39dcfd6372f59f"
server_key_iv: "c7dc77b8e2eafccef704d3b40965fb36"
server_key_version: "encryptionkey@0"
I20260810 23:01:14.264846 17490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:14.293859 17490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:14.297385 17490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:14.297582 17490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:14.299280 17490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:14.299338 17490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:14.299396 17490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:14.299424 17490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:01:14 dist-test-slave-6bbx krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:14.310212 17490 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:36753
I20260810 23:01:14.310616 17700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:36753 every 8 connection(s)
I20260810 23:01:14.311246 17490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:14.313408 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.300167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:41055 (local address 127.15.56.62:39895)
0810 23:01:14.300292 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:14.300310 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:14.301466 (+ 1156us) server_negotiation.cc:407] Connection header received
0810 23:01:14.302536 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:14.302540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:14.302603 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:14.302653 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:14.304169 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.304758 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.305974 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.306324 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.306437 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.309681 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:14.309684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:14.309686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:14.309713 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:14.311262 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.311831 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.311834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.311836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.311936 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.312295 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.312298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.312300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.312489 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:14.312575 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:14.313103 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 23:01:14.313239 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:01:14.314203 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.300798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:39895 (local address 127.15.56.1:41055)
0810 23:01:14.301313 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:14.301360 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:14.302272 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:14.303057 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:14.303064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:14.303430 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:14.304012 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.304029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.304890 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.304893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.305850 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.305862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.306521 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.306527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.306777 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.307535 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:14.307556 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:14.309498 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:14.311386 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.311392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.311409 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.311709 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.312039 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.312042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.312044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.312188 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.312593 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:14.312600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:14.312856 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 23:01:14.313021 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 23:01:14.313371 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":137,"threads_started":1}
I20260810 23:01:14.315446 17701 heartbeater.cc:347] Connected to a master server at 127.15.56.62:39895
I20260810 23:01:14.315738 17701 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:14.315815 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 17490
I20260810 23:01:14.315935 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:14.316190 15584 external_mini_cluster.cc:1428] Setting key 3c96829c804543a74713f6d74958dfb5
I20260810 23:01:14.316555 17701 heartbeater.cc:511] Master 127.15.56.62:39895 requested a full tablet report, sending...
I20260810 23:01:14.318418 17375 ts_manager.cc:194] Registered new tserver with Master: bfa84e13a7c14495b3a496843569dd5e (127.15.56.1:36753)
I20260810 23:01:14.319900 17375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:41055
I20260810 23:01:14.326526 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.318461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:42939 (local address 127.15.56.62:39895)
0810 23:01:14.318578 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:14.318582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:14.318854 (+ 272us) server_negotiation.cc:407] Connection header received
0810 23:01:14.318961 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:14.318964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:14.319007 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:14.319094 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:14.319623 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.320266 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.321473 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.321797 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.321902 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.323208 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:14.323212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:14.323215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:14.323241 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:14.324835 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.325350 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.325352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.325354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.325426 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.325701 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.325703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.325704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.325895 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:14.325973 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:14.326269 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 23:01:14.326389 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":48}
I20260810 23:01:14.326608 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.318394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:39895 (local address 127.15.56.1:42939)
0810 23:01:14.318730 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:14.318774 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:14.318877 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:14.319118 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:14.319121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:14.319343 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:14.319477 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.319485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.320422 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.320424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.321328 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.321341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.321931 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.321935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.322029 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.322637 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:14.322652 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:14.323056 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:14.324966 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.324969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.324972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.325256 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.325502 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.325505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.325507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.325607 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.325990 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:14.325993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:14.326104 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 23:01:14.326232 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:01:14.326384 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:01:14.346340 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:39895
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:14.462879 17719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:14.463188 17719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:14.463236 17719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:14.467290 17719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:14.467373 17719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:14.467413 17719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:01:14.467446 17719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:14.473053 17719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.56.62:39895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:14.474207 17719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:14.475248 17719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:14.485173 17728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:14.485234 17727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:14.485417 17719 server_base.cc:1034] running on GCE node
W20260810 23:01:14.485184 17730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:14.485878 17719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:14.486543 17719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:14.487756 17719 hybrid_clock.cc:648] HybridClock initialized: now 1786402874487747 us; error 32 us; skew 500 ppm
Aug 10 23:01:14 dist-test-slave-6bbx krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:14.490677 17719 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:01:14.491602 17719 webserver.cc:460] Webserver started at http://127.15.56.2:42063/ using document root <none> and password file <none>
I20260810 23:01:14.492204 17719 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:14.492268 17719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:14.492470 17719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:14.494688 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "918b8e8887b04d75a1dcefa1fb52ffbd"
format_stamp: "Formatted at 2026-08-10 23:01:14 on dist-test-slave-6bbx"
server_key: "ce8198c5b387af544406ac2295a848da"
server_key_iv: "3d6feaf7f2c576dd4020e097c5c5421a"
server_key_version: "encryptionkey@0"
I20260810 23:01:14.495256 17719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "918b8e8887b04d75a1dcefa1fb52ffbd"
format_stamp: "Formatted at 2026-08-10 23:01:14 on dist-test-slave-6bbx"
server_key: "ce8198c5b387af544406ac2295a848da"
server_key_iv: "3d6feaf7f2c576dd4020e097c5c5421a"
server_key_version: "encryptionkey@0"
I20260810 23:01:14.499243 17719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:01:14.501876 17740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:14.503182 17719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:14.503281 17719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "918b8e8887b04d75a1dcefa1fb52ffbd"
format_stamp: "Formatted at 2026-08-10 23:01:14 on dist-test-slave-6bbx"
server_key: "ce8198c5b387af544406ac2295a848da"
server_key_iv: "3d6feaf7f2c576dd4020e097c5c5421a"
server_key_version: "encryptionkey@0"
I20260810 23:01:14.503393 17719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:14.517552 17719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:14.520650 17719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:14.520835 17719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:14.522318 17719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:14.522377 17719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:14.522421 17719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:14.522449 17719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:01:14 dist-test-slave-6bbx krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:14.533003 17719 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:40999
I20260810 23:01:14.533100 17917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:40999 every 8 connection(s)
I20260810 23:01:14.533954 17719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:14.536087 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.523244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:43045 (local address 127.15.56.62:39895)
0810 23:01:14.523362 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:14.523366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:14.524488 (+ 1122us) server_negotiation.cc:407] Connection header received
0810 23:01:14.525581 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:14.525584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:14.525646 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:14.525731 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:14.527370 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.528137 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.529336 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.529566 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.529657 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.532410 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:14.532413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:14.532415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:14.532443 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:14.533932 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.534541 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.534545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.534547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.534648 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.535003 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.535006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.535008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.535199 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:14.535284 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:14.535795 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 23:01:14.535923 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:01:14.536885 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.523804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:39895 (local address 127.15.56.2:43045)
0810 23:01:14.524328 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:14.524373 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:14.525354 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:14.525978 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:14.525989 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:14.526462 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:14.527179 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.527202 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.528281 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.528284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.529216 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.529227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.529772 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.529777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.529931 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.530494 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:14.530511 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:14.532230 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:14.534044 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.534050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.534064 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.534420 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.534741 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.534744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.534746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.534920 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.535308 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:14.535315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:14.535549 (+ 234us) client_negotiation.cc:769] Sending connection context
0810 23:01:14.535749 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 23:01:14.536072 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":141,"threads_started":1}
I20260810 23:01:14.538139 17920 heartbeater.cc:347] Connected to a master server at 127.15.56.62:39895
I20260810 23:01:14.538295 17920 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:14.538894 17920 heartbeater.cc:511] Master 127.15.56.62:39895 requested a full tablet report, sending...
I20260810 23:01:14.540048 17374 ts_manager.cc:194] Registered new tserver with Master: 918b8e8887b04d75a1dcefa1fb52ffbd (127.15.56.2:40999)
I20260810 23:01:14.540771 17374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:43045
I20260810 23:01:14.543898 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 17719
I20260810 23:01:14.543991 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:14.544237 15584 external_mini_cluster.cc:1428] Setting key e4abb2ef99ad857e6e2c8608bf8262f0
I20260810 23:01:14.547649 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.540593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:39895 (local address 127.15.56.2:34373)
0810 23:01:14.540853 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:14.540865 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:14.540965 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:14.541209 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:14.541211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:14.541398 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:14.541535 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.541543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.542283 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.542286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.543047 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.543057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.543549 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.543552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.543636 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.544172 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:14.544185 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:14.544550 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:14.546110 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.546113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.546115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.546312 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.546597 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.546600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.546602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.546683 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.547120 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:14.547123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:14.547221 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 23:01:14.547317 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:01:14.547466 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
I20260810 23:01:14.547641 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.540655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:34373 (local address 127.15.56.62:39895)
0810 23:01:14.540767 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:14.540770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:14.540938 (+ 168us) server_negotiation.cc:407] Connection header received
0810 23:01:14.541034 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:14.541037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:14.541080 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:14.541159 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:14.541651 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.542157 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.543152 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.543440 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.543519 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.544683 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:14.544686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:14.544688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:14.544710 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:14.546020 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.546409 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.546412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.546414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.546498 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.546768 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.546771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.546773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.546981 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:14.547070 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:14.547389 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 23:01:14.547489 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:01:14.572482 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:39895
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:14.689939 17932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:14.690236 17932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:14.690285 17932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:14.694717 17932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:14.694813 17932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:14.694856 17932 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:01:14.694890 17932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:14.700843 17932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.56.62:39895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:14.701990 17932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:14.703111 17932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:14.713681 17943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:14.713675 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:14.713675 17938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:14.713819 17932 server_base.cc:1034] running on GCE node
I20260810 23:01:14.714484 17932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:14.715181 17932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:14.716408 17932 hybrid_clock.cc:648] HybridClock initialized: now 1786402874716380 us; error 36 us; skew 500 ppm
Aug 10 23:01:14 dist-test-slave-6bbx krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:14.719918 17932 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:01:14.721000 17932 webserver.cc:460] Webserver started at http://127.15.56.3:42245/ using document root <none> and password file <none>
I20260810 23:01:14.721647 17932 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:14.721740 17932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:14.721963 17932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:14.724237 17932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd"
format_stamp: "Formatted at 2026-08-10 23:01:14 on dist-test-slave-6bbx"
server_key: "09d1c83fc83139f4419bd2749481a9bf"
server_key_iv: "f229a076795db1c0dd6d0b045560bd3d"
server_key_version: "encryptionkey@0"
I20260810 23:01:14.724807 17932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd"
format_stamp: "Formatted at 2026-08-10 23:01:14 on dist-test-slave-6bbx"
server_key: "09d1c83fc83139f4419bd2749481a9bf"
server_key_iv: "f229a076795db1c0dd6d0b045560bd3d"
server_key_version: "encryptionkey@0"
I20260810 23:01:14.729072 17932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:01:14.731846 17952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:14.733121 17932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:14.733247 17932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd"
format_stamp: "Formatted at 2026-08-10 23:01:14 on dist-test-slave-6bbx"
server_key: "09d1c83fc83139f4419bd2749481a9bf"
server_key_iv: "f229a076795db1c0dd6d0b045560bd3d"
server_key_version: "encryptionkey@0"
I20260810 23:01:14.733376 17932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:14.749048 17932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:14.752589 17932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:14.752790 17932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:14.754410 17932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:14.754464 17932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:14.754520 17932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:14.754550 17932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:01:14 dist-test-slave-6bbx krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402874, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:14.765267 17932 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:35841
I20260810 23:01:14.765712 18131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:35841 every 8 connection(s)
I20260810 23:01:14.766305 17932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:14.768751 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.755358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:39569 (local address 127.15.56.62:39895)
0810 23:01:14.755481 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:14.755484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:14.756687 (+ 1203us) server_negotiation.cc:407] Connection header received
0810 23:01:14.757854 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:14.757858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:14.757924 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:14.757987 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:14.759925 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.760515 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.761746 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.762046 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.762161 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.764807 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:14.764811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:14.764814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:14.764845 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:14.766397 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.767069 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.767072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.767075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.767175 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.767559 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.767562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.767563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.767852 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:14.767952 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:14.768491 (+ 539us) server_negotiation.cc:299] Negotiation successful
0810 23:01:14.768605 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:01:14.769639 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.755971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:39895 (local address 127.15.56.3:39569)
0810 23:01:14.756503 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:14.756551 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:14.757574 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:14.758481 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:14.758489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:14.758989 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:14.759758 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.759780 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.760647 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.760651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.761620 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.761631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.762216 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.762221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.762400 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.762982 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:14.762999 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:14.764603 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:14.766521 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.766527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.766543 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.766921 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.767293 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.767297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.767299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.767452 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.767948 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:14.767954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:14.768228 (+ 274us) client_negotiation.cc:769] Sending connection context
0810 23:01:14.768429 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 23:01:14.768705 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":138,"threads_started":1}
I20260810 23:01:14.770021 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 17932
I20260810 23:01:14.770130 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:14.770447 15584 external_mini_cluster.cc:1428] Setting key 23fbe215e21b13de6bb1f85ebeab8395
I20260810 23:01:14.771176 18132 heartbeater.cc:347] Connected to a master server at 127.15.56.62:39895
I20260810 23:01:14.771366 18132 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:14.772014 18132 heartbeater.cc:511] Master 127.15.56.62:39895 requested a full tablet report, sending...
I20260810 23:01:14.773092 17375 ts_manager.cc:194] Registered new tserver with Master: 11418dd4aaab4ac38a0ba5d9422ee4cd (127.15.56.3:35841)
I20260810 23:01:14.773803 17375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:39569
I20260810 23:01:14.780230 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.773558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:55361 (local address 127.15.56.62:39895)
0810 23:01:14.773695 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:14.773699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:14.773876 (+ 177us) server_negotiation.cc:407] Connection header received
0810 23:01:14.773988 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:14.773990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:14.774030 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:14.774103 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:14.774640 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.775175 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.776202 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.776426 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.776516 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.777687 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:14.777690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:14.777692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:14.777715 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:14.778896 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.779253 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.779255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.779256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.779324 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.779518 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.779520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.779521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.779708 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:14.779803 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:14.780016 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 23:01:14.780110 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:01:14.780233 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.773498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:39895 (local address 127.15.56.3:55361)
0810 23:01:14.773793 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:14.773809 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:14.773912 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:14.774144 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:14.774147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:14.774364 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:14.774495 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.774503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.775307 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.775311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.776063 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.776073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.776566 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.776570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.776658 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.777198 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:14.777211 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:14.777552 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:14.778987 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.778990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.778992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.779178 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.779386 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:14.779388 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:14.779389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:14.779458 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:14.779806 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:14.779808 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:14.779891 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 23:01:14.779967 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:01:14.780080 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":109,"threads_started":1}
I20260810 23:01:14.784439 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:01:14 dist-test-slave-6bbx krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402874, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:01:14 dist-test-slave-6bbx krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402874, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:14.810269 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.802381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.15.56.62:39895)
0810 23:01:14.802522 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:14.802526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:14.802538 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:01:14.802579 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:14.802581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:14.802619 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:14.802688 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:14.803381 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.803970 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.804785 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.804998 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.805086 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.807423 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:14.807450 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:14.807452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:14.807483 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:14.808737 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.809120 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.809124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.809125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.809205 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:14.809449 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:14.809453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:14.809455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:14.809638 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:14.809766 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:14.809994 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 23:01:14.810107 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:01:14.813083 17375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:01:14.816210 17375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:14.830940 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.826483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.15.56.1:36753)
0810 23:01:14.826836 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:14.826843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:14.826865 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:01:14.826942 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:14.826947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:14.827139 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:14.827568 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:14.827705 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.828529 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.829724 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.830278 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.830379 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.830538 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:14.830690 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 23:01:14.830786 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":94,"threads_started":1}
I20260810 23:01:14.830941 18164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.826045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:36753 (local address 127.0.0.1:60000)
0810 23:01:14.826503 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:14.826515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:14.826592 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:14.827283 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:14.827286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:14.827330 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:14.827426 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.827433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.828633 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.828637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.829587 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.829598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.830395 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.830398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.830506 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.830545 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 23:01:14.830651 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 23:01:14.830778 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":88,"threads_started":1}
I20260810 23:01:14.831174 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.825749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:35841 (local address 127.0.0.1:38890)
0810 23:01:14.826358 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:14.826372 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:14.826475 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:14.827167 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:14.827170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:14.827188 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:14.827271 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.827277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.828385 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.828389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.829706 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.829717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.830803 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.830807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.830891 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.830912 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:01:14.830963 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 23:01:14.831031 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":11,"spinlock_wait_cycles":8448,"thread_start_us":94,"threads_started":1}
I20260810 23:01:14.831322 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.824984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:40999 (local address 127.0.0.1:56228)
0810 23:01:14.825340 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:14.825390 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:14.825523 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:14.826425 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:14.826432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:14.826471 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:14.826611 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.826617 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.828008 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.828011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.829708 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:14.829717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.830994 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.830997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:14.831073 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.831094 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:01:14.831153 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 23:01:14.831212 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":232,"spinlock_wait_cycles":1664,"thread_start_us":89,"threads_started":1}
I20260810 23:01:14.831755 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.825217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.15.56.2:40999)
0810 23:01:14.825669 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:14.825691 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:14.825716 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:01:14.825826 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:14.825833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:14.826064 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:14.826234 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:14.826735 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.827880 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.829848 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.830664 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.830774 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.830938 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:14.831484 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 23:01:14.831578 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":117,"threads_started":1}
I20260810 23:01:14.831812 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:14.825952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.15.56.3:35841)
0810 23:01:14.826722 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:14.826734 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:14.826758 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:01:14.826856 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:14.826861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:14.827047 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:14.827177 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:14.827377 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.828255 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.829860 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:14.830664 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:14.831372 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:14.831538 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:14.831582 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 23:01:14.831660 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":119,"threads_started":1}
I20260810 23:01:14.834215 17819 tablet_service.cc:1467] Processing CreateTablet for tablet a26e328130d747bab21eaa635002e74c (DEFAULT_TABLE table=test-table [id=e06e24a5b7b34f41b4c451c731bc5b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:14.834513 17590 tablet_service.cc:1467] Processing CreateTablet for tablet a26e328130d747bab21eaa635002e74c (DEFAULT_TABLE table=test-table [id=e06e24a5b7b34f41b4c451c731bc5b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:14.834525 18027 tablet_service.cc:1467] Processing CreateTablet for tablet a26e328130d747bab21eaa635002e74c (DEFAULT_TABLE table=test-table [id=e06e24a5b7b34f41b4c451c731bc5b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:14.835258 17819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26e328130d747bab21eaa635002e74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:14.835551 17590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26e328130d747bab21eaa635002e74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:14.835556 18027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26e328130d747bab21eaa635002e74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:14.842514 18171 tablet_bootstrap.cc:492] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: Bootstrap starting.
I20260810 23:01:14.843961 18172 tablet_bootstrap.cc:492] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: Bootstrap starting.
I20260810 23:01:14.844017 18173 tablet_bootstrap.cc:492] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: Bootstrap starting.
I20260810 23:01:14.845073 18171 tablet_bootstrap.cc:654] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:14.845894 18171 log.cc:824] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:14.846189 18173 tablet_bootstrap.cc:654] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:14.846189 18172 tablet_bootstrap.cc:654] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:14.847096 18173 log.cc:824] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:14.847098 18172 log.cc:824] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:14.848634 18171 tablet_bootstrap.cc:492] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: No bootstrap required, opened a new log
I20260810 23:01:14.848850 18171 ts_tablet_manager.cc:1397] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260810 23:01:14.849125 18173 tablet_bootstrap.cc:492] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: No bootstrap required, opened a new log
I20260810 23:01:14.849125 18172 tablet_bootstrap.cc:492] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: No bootstrap required, opened a new log
I20260810 23:01:14.849320 18172 ts_tablet_manager.cc:1397] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:01:14.849319 18173 ts_tablet_manager.cc:1397] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:01:14.851919 18171 raft_consensus.cc:348] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } }
I20260810 23:01:14.852157 18171 raft_consensus.cc:374] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:14.852212 18171 raft_consensus.cc:729] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918b8e8887b04d75a1dcefa1fb52ffbd, State: Initialized, Role: FOLLOWER
I20260810 23:01:14.852650 18171 consensus_queue.cc:260] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } }
I20260810 23:01:14.853322 17920 heartbeater.cc:503] Master 127.15.56.62:39895 was elected leader, sending a full tablet report...
I20260810 23:01:14.853250 18173 raft_consensus.cc:348] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } }
I20260810 23:01:14.853502 18173 raft_consensus.cc:374] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:14.853546 18173 raft_consensus.cc:729] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11418dd4aaab4ac38a0ba5d9422ee4cd, State: Initialized, Role: FOLLOWER
I20260810 23:01:14.853441 18172 raft_consensus.cc:348] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } }
I20260810 23:01:14.853672 18172 raft_consensus.cc:374] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:14.853729 18172 raft_consensus.cc:729] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa84e13a7c14495b3a496843569dd5e, State: Initialized, Role: FOLLOWER
I20260810 23:01:14.854017 18173 consensus_queue.cc:260] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } }
I20260810 23:01:14.854138 18172 consensus_queue.cc:260] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } }
I20260810 23:01:14.854947 18171 ts_tablet_manager.cc:1428] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:01:14.855057 18172 ts_tablet_manager.cc:1428] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:01:14.855057 18173 ts_tablet_manager.cc:1428] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:01:14.855450 17701 heartbeater.cc:503] Master 127.15.56.62:39895 was elected leader, sending a full tablet report...
I20260810 23:01:14.855893 18132 heartbeater.cc:503] Master 127.15.56.62:39895 was elected leader, sending a full tablet report...
W20260810 23:01:15.017599 18133 tablet.cc:2369] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:01:15.035164 17921 tablet.cc:2369] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:01:15.051905 18181 raft_consensus.cc:482] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:15.052137 18181 raft_consensus.cc:504] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } }
I20260810 23:01:15.053373 18181 leader_election.cc:288] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 918b8e8887b04d75a1dcefa1fb52ffbd (127.15.56.2:40999), bfa84e13a7c14495b3a496843569dd5e (127.15.56.1:36753)
I20260810 23:01:15.057641 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.053690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:40999 (local address 127.15.56.3:50979)
0810 23:01:15.053858 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:15.053876 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:15.053997 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:15.054515 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:15.054518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:15.054534 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:15.054664 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.054671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.055441 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.055443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.056619 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.056631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.057302 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.057304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.057389 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.057412 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:01:15.057470 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 23:01:15.057529 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 23:01:15.057698 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.053789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:50979 (local address 127.15.56.2:40999)
0810 23:01:15.053921 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.053926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.053943 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:01:15.054331 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.054335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.054394 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.054476 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:15.054773 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.055330 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.056737 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.057184 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.057288 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.057366 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.057496 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.057568 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:01:15.058288 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.054001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:36753 (local address 127.15.56.3:44731)
0810 23:01:15.054344 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:15.054356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:15.054454 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:15.054681 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:15.054683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:15.054699 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:15.055145 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.055152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.055955 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.055959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.056969 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.056977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.057900 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.057904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.057998 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.058015 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:01:15.058085 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:01:15.058151 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
I20260810 23:01:15.058317 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.054176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:44731 (local address 127.15.56.1:36753)
0810 23:01:15.054303 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.054313 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.054426 (+ 113us) server_negotiation.cc:407] Connection header received
0810 23:01:15.054537 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.054540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.054612 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.054668 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:15.055261 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.055852 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.057074 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.057660 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.057761 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.057833 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.058118 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.058196 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:01:15.058429 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26e328130d747bab21eaa635002e74c" candidate_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" is_pre_election: true
I20260810 23:01:15.058800 17852 raft_consensus.cc:2454] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11418dd4aaab4ac38a0ba5d9422ee4cd in term 0.
I20260810 23:01:15.059021 17619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26e328130d747bab21eaa635002e74c" candidate_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa84e13a7c14495b3a496843569dd5e" is_pre_election: true
I20260810 23:01:15.059350 17619 raft_consensus.cc:2454] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11418dd4aaab4ac38a0ba5d9422ee4cd in term 0.
I20260810 23:01:15.059360 17953 leader_election.cc:302] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11418dd4aaab4ac38a0ba5d9422ee4cd, 918b8e8887b04d75a1dcefa1fb52ffbd; no voters:
I20260810 23:01:15.059644 18181 raft_consensus.cc:2781] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:15.059775 18181 raft_consensus.cc:482] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:15.059839 18181 raft_consensus.cc:3037] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:15.060907 18181 raft_consensus.cc:504] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } }
I20260810 23:01:15.061236 18181 leader_election.cc:288] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [CANDIDATE]: Term 1 election: Requested vote from peers 918b8e8887b04d75a1dcefa1fb52ffbd (127.15.56.2:40999), bfa84e13a7c14495b3a496843569dd5e (127.15.56.1:36753)
I20260810 23:01:15.061574 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26e328130d747bab21eaa635002e74c" candidate_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd"
I20260810 23:01:15.061648 17619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26e328130d747bab21eaa635002e74c" candidate_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa84e13a7c14495b3a496843569dd5e"
I20260810 23:01:15.061690 17852 raft_consensus.cc:3037] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:15.061779 17619 raft_consensus.cc:3037] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Advancing to term 1
W20260810 23:01:15.062659 17702 tablet.cc:2369] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:01:15.062867 17852 raft_consensus.cc:2454] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11418dd4aaab4ac38a0ba5d9422ee4cd in term 1.
I20260810 23:01:15.062947 17619 raft_consensus.cc:2454] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11418dd4aaab4ac38a0ba5d9422ee4cd in term 1.
I20260810 23:01:15.063174 17953 leader_election.cc:302] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11418dd4aaab4ac38a0ba5d9422ee4cd, 918b8e8887b04d75a1dcefa1fb52ffbd; no voters:
I20260810 23:01:15.063437 18181 raft_consensus.cc:2781] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:15.063633 18181 raft_consensus.cc:686] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 1 LEADER]: Becoming Leader. State: Replica: 11418dd4aaab4ac38a0ba5d9422ee4cd, State: Running, Role: LEADER
I20260810 23:01:15.063978 18181 consensus_queue.cc:237] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } }
I20260810 23:01:15.067221 17374 catalog_manager.cc:5495] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 11418dd4aaab4ac38a0ba5d9422ee4cd (127.15.56.3). New cstate: current_term: 1 leader_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } health_report { overall_health: UNKNOWN } } }
I20260810 23:01:15.129953 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.126243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.15.56.3:35841)
0810 23:01:15.126384 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.126388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.126403 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:01:15.126555 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.126558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.126619 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.126710 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:15.127040 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.127569 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.128517 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.128789 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.128876 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.129163 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:15.129629 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:15.129719 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.129772 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.129829 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:01:15.136121 17619 raft_consensus.cc:1261] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e [term 1 FOLLOWER]: Refusing update from remote peer 11418dd4aaab4ac38a0ba5d9422ee4cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:15.136122 17852 raft_consensus.cc:1261] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd [term 1 FOLLOWER]: Refusing update from remote peer 11418dd4aaab4ac38a0ba5d9422ee4cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:15.136875 18181 consensus_queue.cc:1035] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:15.137145 18186 consensus_queue.cc:1035] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:15.146780 18194 mvcc.cc:205] Tried to move back new op lower bound from 7317106176549744640 to 7317106176264704000. Current Snapshot: MvccSnapshot[applied={T|T < 7317106176549744640}]
I20260810 23:01:15.148068 18193 mvcc.cc:205] Tried to move back new op lower bound from 7317106176549744640 to 7317106176264704000. Current Snapshot: MvccSnapshot[applied={T|T < 7317106176549744640}]
I20260810 23:01:15.157047 17374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:15.157272 17374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:15.160003 17374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af: Sending DeleteTablet for 3 replicas of tablet a26e328130d747bab21eaa635002e74c
I20260810 23:01:15.160956 18027 tablet_service.cc:1514] Processing DeleteTablet for tablet a26e328130d747bab21eaa635002e74c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:15 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:38890
I20260810 23:01:15.161044 17590 tablet_service.cc:1514] Processing DeleteTablet for tablet a26e328130d747bab21eaa635002e74c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:15 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:60000
I20260810 23:01:15.161113 17819 tablet_service.cc:1514] Processing DeleteTablet for tablet a26e328130d747bab21eaa635002e74c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:15 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:56228
I20260810 23:01:15.161609 18210 tablet_replica.cc:330] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: stopping tablet replica
I20260810 23:01:15.161738 18206 tablet_replica.cc:330] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: stopping tablet replica
I20260810 23:01:15.161825 17374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:01:15.161975 18210 raft_consensus.cc:2229] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 23:01:15.162252 17374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:15.162389 18210 raft_consensus.cc:2258] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:15.162403 18206 raft_consensus.cc:2229] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:15.162787 18209 tablet_replica.cc:330] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: stopping tablet replica
I20260810 23:01:15.162981 18206 raft_consensus.cc:2258] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:15.163089 18209 raft_consensus.cc:2229] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:15.163399 18209 raft_consensus.cc:2258] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:15.164786 18206 ts_tablet_manager.cc:1905] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:15.164893 18209 ts_tablet_manager.cc:1905] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:15.167599 18206 ts_tablet_manager.cc:1918] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:15.167713 18209 ts_tablet_manager.cc:1918] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:15.167815 18209 log.cc:1196] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/a26e328130d747bab21eaa635002e74c
I20260810 23:01:15.167829 18206 log.cc:1196] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/a26e328130d747bab21eaa635002e74c
I20260810 23:01:15.168138 18209 ts_tablet_manager.cc:1939] T a26e328130d747bab21eaa635002e74c P 918b8e8887b04d75a1dcefa1fb52ffbd: Deleting consensus metadata
I20260810 23:01:15.168423 17590 tablet_service.cc:1467] Processing CreateTablet for tablet d67272432aba4f46ac881e6a04a808eb (DEFAULT_TABLE table=test-table [id=9419f898c2144587a0ad6c0e7f4ad705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:15.168550 17819 tablet_service.cc:1467] Processing CreateTablet for tablet d67272432aba4f46ac881e6a04a808eb (DEFAULT_TABLE table=test-table [id=9419f898c2144587a0ad6c0e7f4ad705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:15.168713 17590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67272432aba4f46ac881e6a04a808eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:15.168684 18027 tablet_service.cc:1467] Processing CreateTablet for tablet d67272432aba4f46ac881e6a04a808eb (DEFAULT_TABLE table=test-table [id=9419f898c2144587a0ad6c0e7f4ad705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:15.168783 17819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67272432aba4f46ac881e6a04a808eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:15.168906 18027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67272432aba4f46ac881e6a04a808eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:15.170295 18210 ts_tablet_manager.cc:1905] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:15.170994 18171 tablet_bootstrap.cc:492] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd: Bootstrap starting.
I20260810 23:01:15.171202 18173 tablet_bootstrap.cc:492] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd: Bootstrap starting.
I20260810 23:01:15.171201 17350 catalog_manager.cc:4841] TS 918b8e8887b04d75a1dcefa1fb52ffbd (127.15.56.2:40999): tablet a26e328130d747bab21eaa635002e74c (table test-table [id=e06e24a5b7b34f41b4c451c731bc5b8e]) successfully deleted
I20260810 23:01:15.171962 18171 tablet_bootstrap.cc:654] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:15.172119 18173 tablet_bootstrap.cc:654] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:15.172828 18210 ts_tablet_manager.cc:1918] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:15.172924 18210 log.cc:1196] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/wal/wals/a26e328130d747bab21eaa635002e74c
I20260810 23:01:15.173043 18206 ts_tablet_manager.cc:1939] T a26e328130d747bab21eaa635002e74c P 11418dd4aaab4ac38a0ba5d9422ee4cd: Deleting consensus metadata
I20260810 23:01:15.172850 18172 tablet_bootstrap.cc:492] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e: Bootstrap starting.
I20260810 23:01:15.173242 18210 ts_tablet_manager.cc:1939] T a26e328130d747bab21eaa635002e74c P bfa84e13a7c14495b3a496843569dd5e: Deleting consensus metadata
I20260810 23:01:15.173475 18171 tablet_bootstrap.cc:492] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd: No bootstrap required, opened a new log
I20260810 23:01:15.173566 18171 ts_tablet_manager.cc:1397] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:15.174016 18172 tablet_bootstrap.cc:654] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:15.174024 18171 raft_consensus.cc:348] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.174144 18171 raft_consensus.cc:374] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:15.174178 18171 raft_consensus.cc:729] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918b8e8887b04d75a1dcefa1fb52ffbd, State: Initialized, Role: FOLLOWER
I20260810 23:01:15.174811 17350 catalog_manager.cc:4841] TS 11418dd4aaab4ac38a0ba5d9422ee4cd (127.15.56.3:35841): tablet a26e328130d747bab21eaa635002e74c (table test-table [id=e06e24a5b7b34f41b4c451c731bc5b8e]) successfully deleted
I20260810 23:01:15.175156 18171 consensus_queue.cc:260] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.175365 18173 tablet_bootstrap.cc:492] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd: No bootstrap required, opened a new log
I20260810 23:01:15.175439 18173 ts_tablet_manager.cc:1397] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:01:15.175571 18172 tablet_bootstrap.cc:492] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e: No bootstrap required, opened a new log
I20260810 23:01:15.175642 18172 ts_tablet_manager.cc:1397] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:15.175899 18173 raft_consensus.cc:348] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.175995 18171 ts_tablet_manager.cc:1428] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:01:15.176002 18173 raft_consensus.cc:374] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:15.176031 18173 raft_consensus.cc:729] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11418dd4aaab4ac38a0ba5d9422ee4cd, State: Initialized, Role: FOLLOWER
I20260810 23:01:15.176074 18172 raft_consensus.cc:348] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.176170 18172 raft_consensus.cc:374] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:15.176148 18173 consensus_queue.cc:260] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.176200 18172 raft_consensus.cc:729] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa84e13a7c14495b3a496843569dd5e, State: Initialized, Role: FOLLOWER
I20260810 23:01:15.176309 18172 consensus_queue.cc:260] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.176383 18173 ts_tablet_manager.cc:1428] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:15.176545 18172 ts_tablet_manager.cc:1428] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:15.178547 17350 catalog_manager.cc:4841] TS bfa84e13a7c14495b3a496843569dd5e (127.15.56.1:36753): tablet a26e328130d747bab21eaa635002e74c (table test-table [id=e06e24a5b7b34f41b4c451c731bc5b8e]) successfully deleted
I20260810 23:01:15.334383 18182 raft_consensus.cc:482] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:15.334544 18182 raft_consensus.cc:504] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.335731 18182 leader_election.cc:288] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 918b8e8887b04d75a1dcefa1fb52ffbd (127.15.56.2:40999), 11418dd4aaab4ac38a0ba5d9422ee4cd (127.15.56.3:35841)
I20260810 23:01:15.340109 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.336002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:40999 (local address 127.15.56.1:48991)
0810 23:01:15.336338 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:15.336354 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:15.336480 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:15.336817 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:15.336820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:15.336846 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:15.336958 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.336965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.337846 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.337849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.338848 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.338859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.339644 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.339648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.339789 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.339817 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:01:15.339892 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 23:01:15.339964 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260810 23:01:15.340130 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.336104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:48991 (local address 127.15.56.2:40999)
0810 23:01:15.336320 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.336325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.336431 (+ 106us) server_negotiation.cc:407] Connection header received
0810 23:01:15.336604 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.336607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.336665 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.336799 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:15.337082 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.337742 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.339003 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.339534 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.339634 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.339730 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.339919 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.339997 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:01:15.340399 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.336536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:53615 (local address 127.15.56.3:35841)
0810 23:01:15.336669 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.336673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.336824 (+ 151us) server_negotiation.cc:407] Connection header received
0810 23:01:15.336974 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.336977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.337029 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.337091 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:15.337435 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.338136 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.339298 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.339809 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.339917 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.339978 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.340193 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.340277 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:01:15.340415 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.336411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:35841 (local address 127.15.56.1:53615)
0810 23:01:15.336754 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:15.336767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:15.336870 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:15.337120 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:15.337123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:15.337142 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:15.337293 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.337299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.338251 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.338254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.339163 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.339174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.339937 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.339940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.340037 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.340061 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:01:15.340175 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 23:01:15.340245 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":23,"thread_start_us":95,"threads_started":1}
I20260810 23:01:15.340968 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67272432aba4f46ac881e6a04a808eb" candidate_uuid: "bfa84e13a7c14495b3a496843569dd5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" is_pre_election: true
I20260810 23:01:15.341033 18060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67272432aba4f46ac881e6a04a808eb" candidate_uuid: "bfa84e13a7c14495b3a496843569dd5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" is_pre_election: true
I20260810 23:01:15.341120 17852 raft_consensus.cc:2454] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa84e13a7c14495b3a496843569dd5e in term 0.
I20260810 23:01:15.341293 18060 raft_consensus.cc:2454] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfa84e13a7c14495b3a496843569dd5e in term 0.
I20260810 23:01:15.341485 17515 leader_election.cc:302] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 918b8e8887b04d75a1dcefa1fb52ffbd, bfa84e13a7c14495b3a496843569dd5e; no voters:
I20260810 23:01:15.341755 18182 raft_consensus.cc:2781] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:15.341812 18182 raft_consensus.cc:482] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:15.341856 18182 raft_consensus.cc:3037] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:15.342893 18182 raft_consensus.cc:504] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.343223 18182 leader_election.cc:288] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [CANDIDATE]: Term 1 election: Requested vote from peers 918b8e8887b04d75a1dcefa1fb52ffbd (127.15.56.2:40999), 11418dd4aaab4ac38a0ba5d9422ee4cd (127.15.56.3:35841)
I20260810 23:01:15.343567 17852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67272432aba4f46ac881e6a04a808eb" candidate_uuid: "bfa84e13a7c14495b3a496843569dd5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd"
I20260810 23:01:15.343621 18060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67272432aba4f46ac881e6a04a808eb" candidate_uuid: "bfa84e13a7c14495b3a496843569dd5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd"
I20260810 23:01:15.343706 17852 raft_consensus.cc:3037] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:15.343773 18060 raft_consensus.cc:3037] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:15.344633 17852 raft_consensus.cc:2454] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa84e13a7c14495b3a496843569dd5e in term 1.
I20260810 23:01:15.344779 18060 raft_consensus.cc:2454] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfa84e13a7c14495b3a496843569dd5e in term 1.
I20260810 23:01:15.344949 17515 leader_election.cc:302] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 918b8e8887b04d75a1dcefa1fb52ffbd, bfa84e13a7c14495b3a496843569dd5e; no voters:
I20260810 23:01:15.345129 18182 raft_consensus.cc:2781] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:15.345355 18182 raft_consensus.cc:686] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 1 LEADER]: Becoming Leader. State: Replica: bfa84e13a7c14495b3a496843569dd5e, State: Running, Role: LEADER
I20260810 23:01:15.345616 18182 consensus_queue.cc:237] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.348140 17374 catalog_manager.cc:5495] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e reported cstate change: term changed from 0 to 1, leader changed from <none> to bfa84e13a7c14495b3a496843569dd5e (127.15.56.1). New cstate: current_term: 1 leader_uuid: "bfa84e13a7c14495b3a496843569dd5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:01:15 dist-test-slave-6bbx krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402873, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:15.398710 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.389656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:39895 (local address 127.0.0.1:39692)
0810 23:01:15.390027 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:15.390040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:15.390155 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:15.390511 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:15.390514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:15.390983 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:15.391094 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.391101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.391988 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.391991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.392547 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.392554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.393081 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.393084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.393158 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.393789 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:15.393801 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:15.395286 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:15.396975 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:15.396985 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:15.396992 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:15.397228 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:15.397508 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:15.397511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:15.397512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:15.397620 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:15.398078 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:15.398086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:15.398216 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 23:01:15.398331 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 23:01:15.398456 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":127,"threads_started":1}
I20260810 23:01:15.398712 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.389757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.15.56.62:39895)
0810 23:01:15.390014 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.390021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.390111 (+ 90us) server_negotiation.cc:407] Connection header received
0810 23:01:15.390312 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.390315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.390363 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.390458 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:15.391245 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.391882 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.392684 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.392935 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.393025 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.395421 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:15.395425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:15.395427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:15.395449 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:15.396872 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:15.397324 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:15.397326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:15.397327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:15.397401 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:15.397735 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:15.397739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:15.397740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:15.397936 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:15.398045 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.398406 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.398520 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
I20260810 23:01:15.402998 17374 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:39692:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:01:15.403833 17374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:15.408646 17819 tablet_service.cc:1467] Processing CreateTablet for tablet e755135cd0a841f4a0a06055b2ba2dc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2148d76c4be64305b191d31cd0e6f6c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:01:15.408643 17590 tablet_service.cc:1467] Processing CreateTablet for tablet e755135cd0a841f4a0a06055b2ba2dc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2148d76c4be64305b191d31cd0e6f6c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:01:15.408691 18027 tablet_service.cc:1467] Processing CreateTablet for tablet e755135cd0a841f4a0a06055b2ba2dc0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2148d76c4be64305b191d31cd0e6f6c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:01:15.409128 18027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e755135cd0a841f4a0a06055b2ba2dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:15.409127 17590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e755135cd0a841f4a0a06055b2ba2dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:15.409159 17819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e755135cd0a841f4a0a06055b2ba2dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:15.411751 18171 tablet_bootstrap.cc:492] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: Bootstrap starting.
I20260810 23:01:15.411936 18173 tablet_bootstrap.cc:492] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd: Bootstrap starting.
I20260810 23:01:15.411973 18172 tablet_bootstrap.cc:492] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e: Bootstrap starting.
I20260810 23:01:15.412670 18171 tablet_bootstrap.cc:654] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:15.412844 18172 tablet_bootstrap.cc:654] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:15.412838 18173 tablet_bootstrap.cc:654] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:15.414148 18173 tablet_bootstrap.cc:492] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd: No bootstrap required, opened a new log
I20260810 23:01:15.414147 18172 tablet_bootstrap.cc:492] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e: No bootstrap required, opened a new log
I20260810 23:01:15.414237 18173 ts_tablet_manager.cc:1397] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:01:15.414237 18172 ts_tablet_manager.cc:1397] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:01:15.414639 18173 raft_consensus.cc:348] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.414639 18172 raft_consensus.cc:348] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.414763 18173 raft_consensus.cc:374] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:15.414763 18172 raft_consensus.cc:374] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:15.414801 18173 raft_consensus.cc:729] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11418dd4aaab4ac38a0ba5d9422ee4cd, State: Initialized, Role: FOLLOWER
I20260810 23:01:15.414808 18172 raft_consensus.cc:729] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfa84e13a7c14495b3a496843569dd5e, State: Initialized, Role: FOLLOWER
I20260810 23:01:15.414929 18172 consensus_queue.cc:260] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.414933 18173 consensus_queue.cc:260] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.415200 18173 ts_tablet_manager.cc:1428] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:15.415200 18172 ts_tablet_manager.cc:1428] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:15.415352 18171 tablet_bootstrap.cc:492] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: No bootstrap required, opened a new log
I20260810 23:01:15.415441 18171 ts_tablet_manager.cc:1397] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 23:01:15.415469 18182 tablet_replica.cc:439] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } }
I20260810 23:01:15.415472 18186 tablet_replica.cc:439] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } }
I20260810 23:01:15.415633 18182 tablet_replica.cc:442] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:15.415900 18171 raft_consensus.cc:348] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.415997 18171 raft_consensus.cc:374] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:15.416028 18171 raft_consensus.cc:729] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918b8e8887b04d75a1dcefa1fb52ffbd, State: Initialized, Role: FOLLOWER
I20260810 23:01:15.415633 18186 tablet_replica.cc:442] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:15.416152 18171 consensus_queue.cc:260] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.416402 18171 ts_tablet_manager.cc:1428] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:15.416798 18180 tablet_replica.cc:439] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } }
I20260810 23:01:15.416937 18180 tablet_replica.cc:442] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:15.674065 18180 raft_consensus.cc:482] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:15.674223 18180 raft_consensus.cc:504] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.675344 18180 leader_election.cc:288] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfa84e13a7c14495b3a496843569dd5e (127.15.56.1:36753), 11418dd4aaab4ac38a0ba5d9422ee4cd (127.15.56.3:35841)
I20260810 23:01:15.679591 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.676168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:51503 (local address 127.15.56.3:35841)
0810 23:01:15.676293 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.676298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.676527 (+ 229us) server_negotiation.cc:407] Connection header received
0810 23:01:15.676587 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.676590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.676654 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.676724 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:15.676992 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.677518 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.678581 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.679041 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.679129 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.679185 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.679380 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.679457 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:01:15.679652 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.675753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:34477 (local address 127.15.56.1:36753)
0810 23:01:15.676010 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.676014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.676033 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:01:15.676172 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.676176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.676231 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.676325 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:15.676658 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.677318 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.678605 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.679127 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.679236 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.679299 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.679452 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.679534 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260810 23:01:15.679637 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.675623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:36753 (local address 127.15.56.2:34477)
0810 23:01:15.675892 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:15.675906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:15.676024 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:15.676370 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:15.676374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:15.676399 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:15.676533 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.676540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.677434 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.677438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.678479 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.678490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.679248 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.679250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.679327 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.679351 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:01:15.679426 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 23:01:15.679492 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":34,"thread_start_us":100,"threads_started":1}
I20260810 23:01:15.679613 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.676045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:35841 (local address 127.15.56.2:51503)
0810 23:01:15.676429 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:15.676441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:15.676521 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:15.676744 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:15.676747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:15.676763 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:15.676879 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.676886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.677629 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.677632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.678466 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.678475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.679159 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.679164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.679260 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.679284 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:01:15.679362 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:01:15.679430 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":35,"thread_start_us":52,"threads_started":1}
I20260810 23:01:15.680188 17619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e755135cd0a841f4a0a06055b2ba2dc0" candidate_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa84e13a7c14495b3a496843569dd5e" is_pre_election: true
I20260810 23:01:15.680320 18060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e755135cd0a841f4a0a06055b2ba2dc0" candidate_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" is_pre_election: true
I20260810 23:01:15.680367 17619 raft_consensus.cc:2454] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 918b8e8887b04d75a1dcefa1fb52ffbd in term 0.
I20260810 23:01:15.680459 18060 raft_consensus.cc:2454] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 918b8e8887b04d75a1dcefa1fb52ffbd in term 0.
I20260810 23:01:15.680745 17741 leader_election.cc:302] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 918b8e8887b04d75a1dcefa1fb52ffbd, bfa84e13a7c14495b3a496843569dd5e; no voters:
I20260810 23:01:15.681000 18180 raft_consensus.cc:2781] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:15.681061 18180 raft_consensus.cc:482] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:15.681098 18180 raft_consensus.cc:3037] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:15.682227 18180 raft_consensus.cc:504] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.682559 18180 leader_election.cc:288] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [CANDIDATE]: Term 1 election: Requested vote from peers bfa84e13a7c14495b3a496843569dd5e (127.15.56.1:36753), 11418dd4aaab4ac38a0ba5d9422ee4cd (127.15.56.3:35841)
I20260810 23:01:15.682915 17619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e755135cd0a841f4a0a06055b2ba2dc0" candidate_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfa84e13a7c14495b3a496843569dd5e"
I20260810 23:01:15.683036 17619 raft_consensus.cc:3037] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:15.683017 18060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e755135cd0a841f4a0a06055b2ba2dc0" candidate_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd"
I20260810 23:01:15.683184 18060 raft_consensus.cc:3037] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:15.683984 17619 raft_consensus.cc:2454] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 918b8e8887b04d75a1dcefa1fb52ffbd in term 1.
I20260810 23:01:15.684214 18060 raft_consensus.cc:2454] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 918b8e8887b04d75a1dcefa1fb52ffbd in term 1.
I20260810 23:01:15.684242 17741 leader_election.cc:302] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 918b8e8887b04d75a1dcefa1fb52ffbd, bfa84e13a7c14495b3a496843569dd5e; no voters:
I20260810 23:01:15.684410 18180 raft_consensus.cc:2781] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:15.684579 18180 raft_consensus.cc:686] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [term 1 LEADER]: Becoming Leader. State: Replica: 918b8e8887b04d75a1dcefa1fb52ffbd, State: Running, Role: LEADER
I20260810 23:01:15.684829 18180 consensus_queue.cc:237] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } }
I20260810 23:01:15.685926 18241 tablet_replica.cc:439] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: TxnStatusTablet state changed. Reason: New leader 918b8e8887b04d75a1dcefa1fb52ffbd. Latest consensus state: current_term: 1 leader_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } }
I20260810 23:01:15.686054 18241 tablet_replica.cc:442] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:01:15.686630 18245 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:01:15.686805 18245 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:01:15.687336 17375 catalog_manager.cc:5495] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 918b8e8887b04d75a1dcefa1fb52ffbd (127.15.56.2). New cstate: current_term: 1 leader_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } health_report { overall_health: UNKNOWN } } }
I20260810 23:01:15.718379 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.714195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:40999 (local address 127.0.0.1:56230)
0810 23:01:15.714354 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:15.714370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:15.714484 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:15.714832 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:15.714834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:15.714842 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:01:15.714940 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.714944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.715954 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.715957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.716619 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.716629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.717243 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.717245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.717311 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.717358 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:01:15.718131 (+ 773us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:01:15.718139 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:01:15.718175 (+ 36us) client_negotiation.cc:240] Negotiation successful
0810 23:01:15.718260 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 23:01:15.718397 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.714340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.15.56.2:40999)
0810 23:01:15.714480 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.714485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.714501 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:01:15.714641 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.714644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.714714 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.714829 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:15.715030 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.715854 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.716729 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.717079 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.717233 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.717484 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:15.718047 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:15.718174 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.718206 (+ 32us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.718261 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:01:15.721567 17619 raft_consensus.cc:1261] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e [term 1 FOLLOWER]: Refusing update from remote peer 918b8e8887b04d75a1dcefa1fb52ffbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:15.721879 18060 raft_consensus.cc:1261] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 1 FOLLOWER]: Refusing update from remote peer 918b8e8887b04d75a1dcefa1fb52ffbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:15.722239 18241 consensus_queue.cc:1035] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:15.722438 18180 consensus_queue.cc:1035] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:15.724032 18182 tablet_replica.cc:439] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e: TxnStatusTablet state changed. Reason: New leader 918b8e8887b04d75a1dcefa1fb52ffbd. Latest consensus state: current_term: 1 leader_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } }
I20260810 23:01:15.724152 18182 tablet_replica.cc:442] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:15.724339 18186 tablet_replica.cc:439] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd: TxnStatusTablet state changed. Reason: New leader 918b8e8887b04d75a1dcefa1fb52ffbd. Latest consensus state: current_term: 1 leader_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } }
I20260810 23:01:15.724462 18186 tablet_replica.cc:442] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:15.727221 18180 tablet_replica.cc:439] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } }
I20260810 23:01:15.727273 18186 tablet_replica.cc:439] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } }
I20260810 23:01:15.727347 18180 tablet_replica.cc:442] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:01:15.727382 18186 tablet_replica.cc:442] T e755135cd0a841f4a0a06055b2ba2dc0 P 11418dd4aaab4ac38a0ba5d9422ee4cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:15.727353 18182 tablet_replica.cc:439] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } }
I20260810 23:01:15.727444 18182 tablet_replica.cc:442] T e755135cd0a841f4a0a06055b2ba2dc0 P bfa84e13a7c14495b3a496843569dd5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:15.727468 18180 tablet_replica.cc:439] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfa84e13a7c14495b3a496843569dd5e" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 36753 } } peers { permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 } } peers { permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 } } }
I20260810 23:01:15.727540 18180 tablet_replica.cc:442] T e755135cd0a841f4a0a06055b2ba2dc0 P 918b8e8887b04d75a1dcefa1fb52ffbd: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:01:15.728924 18251 mvcc.cc:205] Tried to move back new op lower bound from 7317106178950758400 to 7317106178807480320. Current Snapshot: MvccSnapshot[applied={T|T < 7317106178950758400 or (T in {7317106178950758400})}]
I20260810 23:01:15.738509 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.734350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.15.56.1:36753)
0810 23:01:15.734473 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.734478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.734525 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:01:15.734690 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.734694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.734764 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.734878 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:15.735087 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.735907 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.736786 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.737157 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.737257 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.737487 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:15.738089 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:15.738215 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.738295 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.738360 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:01:15.750234 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.746729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:59925 (local address 127.15.56.2:40999)
0810 23:01:15.746875 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.746879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.747160 (+ 281us) server_negotiation.cc:407] Connection header received
0810 23:01:15.747308 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.747311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.747359 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.747427 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:15.747713 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.748311 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.749152 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.749388 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.749476 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.749780 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:15.749862 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:15.749920 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.750020 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.750077 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:01:15.750244 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.746640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:40999 (local address 127.15.56.1:59925)
0810 23:01:15.747064 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:15.747078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:15.747183 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:15.747460 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:15.747463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:15.747471 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:01:15.747593 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.747598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.748420 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.748424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.749039 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.749047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.749493 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.749496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.749585 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.749638 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:01:15.749937 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:01:15.749944 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:01:15.750003 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 23:01:15.750097 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":119,"threads_started":1}
I20260810 23:01:15.759181 17852 raft_consensus.cc:1261] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd [term 1 FOLLOWER]: Refusing update from remote peer bfa84e13a7c14495b3a496843569dd5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:15.759248 18060 raft_consensus.cc:1261] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 1 FOLLOWER]: Refusing update from remote peer bfa84e13a7c14495b3a496843569dd5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:15.759721 18182 consensus_queue.cc:1035] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "918b8e8887b04d75a1dcefa1fb52ffbd" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 40999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:15.760190 18219 consensus_queue.cc:1035] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "11418dd4aaab4ac38a0ba5d9422ee4cd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:15.790989 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.786619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:44563 (local address 127.15.56.1:36753)
0810 23:01:15.786755 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.786759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.787048 (+ 289us) server_negotiation.cc:407] Connection header received
0810 23:01:15.787179 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.787182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.787243 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.787325 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:15.787618 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.788386 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.789708 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.790062 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.790160 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.790505 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:15.790596 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:15.790653 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.790773 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.790831 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:01:15.791003 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.786504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:36753 (local address 127.15.56.2:44563)
0810 23:01:15.786946 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:15.786961 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:15.787074 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:15.787355 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:15.787358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:15.787367 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:01:15.787484 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.787490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.788664 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.788667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.789576 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:15.789589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.790189 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.790193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:15.790289 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.790358 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:01:15.790681 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:01:15.790687 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:01:15.790759 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 23:01:15.790854 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":104,"threads_started":1}
I20260810 23:01:15.820122 17373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:15.820344 17373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:15.822988 17373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4579d60920e845b28fd5bffc47bbf2af: Sending DeleteTablet for 3 replicas of tablet d67272432aba4f46ac881e6a04a808eb
I20260810 23:01:15.823997 18027 tablet_service.cc:1514] Processing DeleteTablet for tablet d67272432aba4f46ac881e6a04a808eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:15 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:38890
I20260810 23:01:15.824280 17590 tablet_service.cc:1514] Processing DeleteTablet for tablet d67272432aba4f46ac881e6a04a808eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:15 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:60000
I20260810 23:01:15.824460 18284 tablet_replica.cc:330] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd: stopping tablet replica
I20260810 23:01:15.824597 18284 raft_consensus.cc:2229] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:15.824764 18284 raft_consensus.cc:2258] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:15.824816 17819 tablet_service.cc:1514] Processing DeleteTablet for tablet d67272432aba4f46ac881e6a04a808eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:15 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:56228
I20260810 23:01:15.825640 18284 ts_tablet_manager.cc:1905] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:15.826421 18285 tablet_replica.cc:330] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e: stopping tablet replica
I20260810 23:01:15.826632 18285 raft_consensus.cc:2229] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:15.827059 18285 raft_consensus.cc:2258] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:15.827348 18286 tablet_replica.cc:330] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd: stopping tablet replica
I20260810 23:01:15.827632 18286 raft_consensus.cc:2229] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:15.828533 18285 ts_tablet_manager.cc:1905] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:15.828773 18286 raft_consensus.cc:2258] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:15.829053 18284 ts_tablet_manager.cc:1918] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:15.829208 18284 log.cc:1196] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/d67272432aba4f46ac881e6a04a808eb
I20260810 23:01:15.829548 18284 ts_tablet_manager.cc:1939] T d67272432aba4f46ac881e6a04a808eb P 11418dd4aaab4ac38a0ba5d9422ee4cd: Deleting consensus metadata
I20260810 23:01:15.829576 18286 ts_tablet_manager.cc:1905] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:15.830660 17350 catalog_manager.cc:4841] TS 11418dd4aaab4ac38a0ba5d9422ee4cd (127.15.56.3:35841): tablet d67272432aba4f46ac881e6a04a808eb (table test-table [id=9419f898c2144587a0ad6c0e7f4ad705]) successfully deleted
I20260810 23:01:15.832911 18285 ts_tablet_manager.cc:1918] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:15.833007 18285 log.cc:1196] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-0/wal/wals/d67272432aba4f46ac881e6a04a808eb
I20260810 23:01:15.836018 18286 ts_tablet_manager.cc:1918] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:15.836150 18286 log.cc:1196] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/d67272432aba4f46ac881e6a04a808eb
I20260810 23:01:15.836477 18285 ts_tablet_manager.cc:1939] T d67272432aba4f46ac881e6a04a808eb P bfa84e13a7c14495b3a496843569dd5e: Deleting consensus metadata
I20260810 23:01:15.836781 18286 ts_tablet_manager.cc:1939] T d67272432aba4f46ac881e6a04a808eb P 918b8e8887b04d75a1dcefa1fb52ffbd: Deleting consensus metadata
I20260810 23:01:15.837630 17350 catalog_manager.cc:4841] TS bfa84e13a7c14495b3a496843569dd5e (127.15.56.1:36753): tablet d67272432aba4f46ac881e6a04a808eb (table test-table [id=9419f898c2144587a0ad6c0e7f4ad705]) successfully deleted
I20260810 23:01:15.838581 17350 catalog_manager.cc:4841] TS 918b8e8887b04d75a1dcefa1fb52ffbd (127.15.56.2:40999): tablet d67272432aba4f46ac881e6a04a808eb (table test-table [id=9419f898c2144587a0ad6c0e7f4ad705]) successfully deleted
Aug 10 23:01:15 dist-test-slave-6bbx krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402874, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:01:15.848640 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.831315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60030 (local address 127.15.56.1:36753)
0810 23:01:15.831429 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.831434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.836021 (+ 4587us) server_negotiation.cc:407] Connection header received
0810 23:01:15.836247 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.836251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.836303 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.836353 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:15.837484 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.838363 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.839764 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.840113 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.840222 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.844218 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:15.844232 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:15.844251 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:15.844285 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:15.846428 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:15.846974 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:15.846981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:15.846983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:15.847083 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:15.847422 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:15.847425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:15.847427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:15.847809 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:15.848045 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.848320 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.848452 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260810 23:01:15.849119 17688 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60030
I20260810 23:01:15.860020 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:15.850987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.15.56.62:39895)
0810 23:01:15.851133 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:15.851138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:15.851512 (+ 374us) server_negotiation.cc:407] Connection header received
0810 23:01:15.851624 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:15.851627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:15.851751 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:15.851825 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:15.852612 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.853460 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.854592 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:15.854958 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:15.855055 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:15.856601 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:15.856605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:15.856608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:15.856637 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:15.858250 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:15.858704 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:15.858708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:15.858710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:15.858805 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:15.859049 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:15.859052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:15.859053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:15.859285 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:15.859420 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:15.859649 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 23:01:15.859833 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 23:01:15.860745 17373 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39698
I20260810 23:01:15.862236 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 17490
I20260810 23:01:15.875070 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 17719
I20260810 23:01:15.893689 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 17932
I20260810 23:01:15.910672 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 17322
2026-08-10T23:01:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3399 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:15 dist-test-slave-6bbx krb5kdc[18296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:15 dist-test-slave-6bbx krb5kdc[18296](info): set up 2 sockets
Aug 10 23:01:15 dist-test-slave-6bbx krb5kdc[18296](info): commencing operation
krb5kdc: starting...
W20260810 23:01:17.074498 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:17 dist-test-slave-6bbx krb5kdc[18296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:17.201053 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:41521
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:41521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:17.318794 18320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:17.319098 18320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:17.319141 18320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:17.323277 18320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:17.323359 18320 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:01:17.323374 18320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:17.323391 18320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:17.323408 18320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:17.323422 18320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:17.329056 18320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:41521
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:41521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:17.330216 18320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:17.331282 18320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:17.340597 18326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:17.340718 18320 server_base.cc:1034] running on GCE node
W20260810 23:01:17.340597 18329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:17.340627 18327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:17.341585 18320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:17.342880 18320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:17.344038 18320 hybrid_clock.cc:648] HybridClock initialized: now 1786402877344019 us; error 34 us; skew 500 ppm
Aug 10 23:01:17 dist-test-slave-6bbx krb5kdc[18296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402877, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:17.346841 18320 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:17.347854 18320 webserver.cc:460] Webserver started at http://127.15.56.62:36823/ using document root <none> and password file <none>
I20260810 23:01:17.348449 18320 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:17.348543 18320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:17.348814 18320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:17.350926 18320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "675a6643931244b3af073a82cbb8d850"
format_stamp: "Formatted at 2026-08-10 23:01:17 on dist-test-slave-6bbx"
server_key: "d407893f302709cee1701fc1f6d7e5b3"
server_key_iv: "7be7572f97718bf113940deeb26e09c5"
server_key_version: "encryptionkey@0"
I20260810 23:01:17.351501 18320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "675a6643931244b3af073a82cbb8d850"
format_stamp: "Formatted at 2026-08-10 23:01:17 on dist-test-slave-6bbx"
server_key: "d407893f302709cee1701fc1f6d7e5b3"
server_key_iv: "7be7572f97718bf113940deeb26e09c5"
server_key_version: "encryptionkey@0"
I20260810 23:01:17.355413 18320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:01:17.358003 18342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:17.359324 18320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:01:17.359458 18320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "675a6643931244b3af073a82cbb8d850"
format_stamp: "Formatted at 2026-08-10 23:01:17 on dist-test-slave-6bbx"
server_key: "d407893f302709cee1701fc1f6d7e5b3"
server_key_iv: "7be7572f97718bf113940deeb26e09c5"
server_key_version: "encryptionkey@0"
I20260810 23:01:17.359563 18320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:17.370189 18320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:17.373404 18320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:17.373597 18320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:17.381778 18320 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:41521
I20260810 23:01:17.381791 18444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:41521 every 8 connection(s)
I20260810 23:01:17.382809 18320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:17.385779 18445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:17.389336 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 18320
I20260810 23:01:17.389457 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:17.389742 15584 external_mini_cluster.cc:1428] Setting key fe2da3151a0d23e4cb5a35ebdcfdcf99
I20260810 23:01:17.392119 18445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850: Bootstrap starting.
I20260810 23:01:17.394644 18445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:17.395462 18445 log.cc:824] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:17.397993 18445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850: No bootstrap required, opened a new log
Aug 10 23:01:17 dist-test-slave-6bbx krb5kdc[18296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:17.401898 18445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675a6643931244b3af073a82cbb8d850" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41521 } }
I20260810 23:01:17.402287 18445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:17.402374 18445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 675a6643931244b3af073a82cbb8d850, State: Initialized, Role: FOLLOWER
I20260810 23:01:17.402917 18445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675a6643931244b3af073a82cbb8d850" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41521 } }
I20260810 23:01:17.403071 18445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:17.403144 18445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:17.403252 18445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:17.404667 18445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675a6643931244b3af073a82cbb8d850" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41521 } }
I20260810 23:01:17.405074 18445 leader_election.cc:302] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 675a6643931244b3af073a82cbb8d850; no voters:
I20260810 23:01:17.405447 18445 leader_election.cc:288] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:17.405499 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:17.391211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.15.56.62:41521)
0810 23:01:17.391651 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:17.391696 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:17.391749 (+ 53us) server_negotiation.cc:407] Connection header received
0810 23:01:17.392419 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:17.392440 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:17.392797 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:17.393091 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:17.393987 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.395107 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.396235 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.396767 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.396911 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:17.399929 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:17.399935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:17.399951 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:17.399981 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:17.401746 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:17.402343 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:17.402351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:17.402359 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:17.402518 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:17.402847 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:17.402850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:17.402851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:17.403412 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:17.403575 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:17.404141 (+ 566us) server_negotiation.cc:299] Negotiation successful
0810 23:01:17.404537 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":123,"threads_started":1}
I20260810 23:01:17.405575 18450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:17.405875 18450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [term 1 LEADER]: Becoming Leader. State: Replica: 675a6643931244b3af073a82cbb8d850, State: Running, Role: LEADER
I20260810 23:01:17.406286 18450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675a6643931244b3af073a82cbb8d850" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41521 } }
I20260810 23:01:17.407009 18445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:17.407737 18450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 675a6643931244b3af073a82cbb8d850. Latest consensus state: current_term: 1 leader_uuid: "675a6643931244b3af073a82cbb8d850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675a6643931244b3af073a82cbb8d850" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41521 } } }
I20260810 23:01:17.407882 18450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:17.408355 18458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:17.408950 18451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "675a6643931244b3af073a82cbb8d850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "675a6643931244b3af073a82cbb8d850" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41521 } } }
I20260810 23:01:17.409150 18451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:17.411962 18458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:17.417919 18458 catalog_manager.cc:1347] Generated new cluster ID: 82dab90fcdbe4874b9bd543fe11799fd
I20260810 23:01:17.417999 18458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:17.424090 18458 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:17.425428 18458 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:17.438684 18458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850: Generated new TSK 0
I20260810 23:01:17.439553 18458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:01:17.475157 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41521
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:17.598495 18479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:17.598830 18479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:17.598879 18479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:17.603192 18479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:17.603276 18479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:17.603319 18479 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:01:17.603332 18479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:01:17.603364 18479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:17.609242 18479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.56.62:41521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:17.610446 18479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:17.611546 18479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:01:17.622444 18479 server_base.cc:1034] running on GCE node
W20260810 23:01:17.622396 18486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:17.622728 18492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:17.622394 18487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:17.623306 18479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:17.624096 18479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:17.625270 18479 hybrid_clock.cc:648] HybridClock initialized: now 1786402877625249 us; error 38 us; skew 500 ppm
Aug 10 23:01:17 dist-test-slave-6bbx krb5kdc[18296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402877, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:17.628342 18479 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:01:17.629318 18479 webserver.cc:460] Webserver started at http://127.15.56.1:42009/ using document root <none> and password file <none>
I20260810 23:01:17.629906 18479 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:17.629979 18479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:17.630218 18479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:17.632474 18479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "df4a4476bbd3417190567a6c72e00c5a"
format_stamp: "Formatted at 2026-08-10 23:01:17 on dist-test-slave-6bbx"
server_key: "e73a97dd864fe42285f991a88288f651"
server_key_iv: "40f16cb4f6fa25142560a63ed3fccc76"
server_key_version: "encryptionkey@0"
I20260810 23:01:17.633068 18479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "df4a4476bbd3417190567a6c72e00c5a"
format_stamp: "Formatted at 2026-08-10 23:01:17 on dist-test-slave-6bbx"
server_key: "e73a97dd864fe42285f991a88288f651"
server_key_iv: "40f16cb4f6fa25142560a63ed3fccc76"
server_key_version: "encryptionkey@0"
I20260810 23:01:17.637367 18479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:01:17.640311 18504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:17.641741 18479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:17.641881 18479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "df4a4476bbd3417190567a6c72e00c5a"
format_stamp: "Formatted at 2026-08-10 23:01:17 on dist-test-slave-6bbx"
server_key: "e73a97dd864fe42285f991a88288f651"
server_key_iv: "40f16cb4f6fa25142560a63ed3fccc76"
server_key_version: "encryptionkey@0"
I20260810 23:01:17.642006 18479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:17.661680 18479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:17.665302 18479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:17.665504 18479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:17.667119 18479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:17.667172 18479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:17.667229 18479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:17.667259 18479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:01:17 dist-test-slave-6bbx krb5kdc[18296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402877, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:17.677728 18479 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:45281
I20260810 23:01:17.678071 18699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:45281 every 8 connection(s)
I20260810 23:01:17.678740 18479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:17.680871 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:17.667974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:53019 (local address 127.15.56.62:41521)
0810 23:01:17.668122 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:17.668126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:17.669094 (+ 968us) server_negotiation.cc:407] Connection header received
0810 23:01:17.670222 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:17.670226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:17.670285 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:17.670368 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:17.672137 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.672941 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.674152 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.674395 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.674485 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:17.677179 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:17.677182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:17.677184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:17.677211 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:17.678677 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:17.679294 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:17.679298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:17.679300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:17.679404 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:17.679762 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:17.679766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:17.679768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:17.679963 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:17.680053 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:17.680565 (+ 512us) server_negotiation.cc:299] Negotiation successful
0810 23:01:17.680702 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:01:17.681639 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:17.668441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41521 (local address 127.15.56.1:53019)
0810 23:01:17.668916 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:17.668968 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:17.669948 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:17.670635 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:17.670646 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:17.671162 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:17.671931 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:17.671956 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.673062 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.673065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:17.674029 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:17.674039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.674543 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.674547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:17.674710 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:17.675354 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:17.675370 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:17.676993 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:17.678795 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:17.678801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:17.678816 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:17.679172 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:17.679485 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:17.679488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:17.679490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:17.679657 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:17.680075 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:17.680082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:17.680324 (+ 242us) client_negotiation.cc:769] Sending connection context
0810 23:01:17.680484 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 23:01:17.680816 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260810 23:01:17.682914 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 18479
I20260810 23:01:17.683023 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:17.683151 18702 heartbeater.cc:347] Connected to a master server at 127.15.56.62:41521
I20260810 23:01:17.683292 18702 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:17.683322 15584 external_mini_cluster.cc:1428] Setting key cd10bdf7ac65ce08afd3bb82a8a2dc7b
I20260810 23:01:17.683900 18702 heartbeater.cc:511] Master 127.15.56.62:41521 requested a full tablet report, sending...
I20260810 23:01:17.685631 18370 ts_manager.cc:194] Registered new tserver with Master: df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281)
I20260810 23:01:17.686800 18370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:53019
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:01:17.694054 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:17.685493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41521 (local address 127.15.56.1:38379)
0810 23:01:17.685784 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:17.685798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:17.685888 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:17.687589 (+ 1701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:17.687592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:17.687853 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:17.687989 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:17.687998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.688980 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.688983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:17.689582 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:17.689591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.690025 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.690028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:17.690120 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:17.690590 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:17.690609 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:17.690922 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:17.692544 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:17.692547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:17.692549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:17.692759 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:17.693020 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:17.693032 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:17.693034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:17.693117 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:17.693505 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:17.693508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:17.693609 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 23:01:17.693699 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:01:17.693858 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":8,"thread_start_us":105,"threads_started":1}
I20260810 23:01:17.694042 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:17.685538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:38379 (local address 127.15.56.62:41521)
0810 23:01:17.687398 (+ 1860us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:17.687401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:17.687414 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:01:17.687447 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:17.687449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:17.687483 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:17.687546 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:17.688114 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.688836 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.689711 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.689921 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.690007 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:17.691057 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:17.691060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:17.691062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:17.691083 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:17.692436 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:17.692849 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:17.692852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:17.692854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:17.692933 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:17.693200 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:17.693203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:17.693205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:17.693415 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:17.693503 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:17.693770 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 23:01:17.693885 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1788}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:01:17.712690 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41521
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:17.830468 18720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:17.830765 18720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:17.830823 18720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:17.834906 18720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:17.834973 18720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:17.835016 18720 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:01:17.835029 18720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:01:17.835078 18720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:17.840505 18720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.56.62:41521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:17.841634 18720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:17.842769 18720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:17.852900 18730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:17.852900 18733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:17.852949 18729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:17.853258 18720 server_base.cc:1034] running on GCE node
I20260810 23:01:17.853686 18720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:17.854316 18720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:17.855495 18720 hybrid_clock.cc:648] HybridClock initialized: now 1786402877855474 us; error 31 us; skew 500 ppm
Aug 10 23:01:17 dist-test-slave-6bbx krb5kdc[18296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402877, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:17.858573 18720 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:01:17.859560 18720 webserver.cc:460] Webserver started at http://127.15.56.2:46819/ using document root <none> and password file <none>
I20260810 23:01:17.860172 18720 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:17.860237 18720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:17.860411 18720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:17.862509 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "f8adb8adeb1f43b992722da92bcce13d"
format_stamp: "Formatted at 2026-08-10 23:01:17 on dist-test-slave-6bbx"
server_key: "58c030f94c47122a0ccf0f94fdeb559f"
server_key_iv: "18d94cf38323fd0ed5b8d8c6052b094d"
server_key_version: "encryptionkey@0"
I20260810 23:01:17.863075 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "f8adb8adeb1f43b992722da92bcce13d"
format_stamp: "Formatted at 2026-08-10 23:01:17 on dist-test-slave-6bbx"
server_key: "58c030f94c47122a0ccf0f94fdeb559f"
server_key_iv: "18d94cf38323fd0ed5b8d8c6052b094d"
server_key_version: "encryptionkey@0"
I20260810 23:01:17.867100 18720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:01:17.869773 18744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:17.871104 18720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:17.871222 18720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "f8adb8adeb1f43b992722da92bcce13d"
format_stamp: "Formatted at 2026-08-10 23:01:17 on dist-test-slave-6bbx"
server_key: "58c030f94c47122a0ccf0f94fdeb559f"
server_key_iv: "18d94cf38323fd0ed5b8d8c6052b094d"
server_key_version: "encryptionkey@0"
I20260810 23:01:17.871325 18720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:17.886116 18720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:17.889446 18720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:17.889626 18720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:17.891260 18720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:17.891327 18720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:17.891384 18720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:17.891413 18720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:01:17 dist-test-slave-6bbx krb5kdc[18296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402877, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:17.902534 18720 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:34973
I20260810 23:01:17.902598 18957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:34973 every 8 connection(s)
I20260810 23:01:17.903874 18720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:17.906241 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:17.892306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:41839 (local address 127.15.56.62:41521)
0810 23:01:17.892436 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:17.892439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:17.893622 (+ 1183us) server_negotiation.cc:407] Connection header received
0810 23:01:17.894849 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:17.894854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:17.894919 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:17.895018 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:17.896544 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.897374 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.898794 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.899032 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.899134 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:17.902006 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:17.902009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:17.902011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:17.902042 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:17.903956 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:17.904673 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:17.904678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:17.904681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:17.904813 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:17.905198 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:17.905201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:17.905202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:17.905416 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:17.905501 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:17.905994 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 23:01:17.906109 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:01:17.907145 18768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:17.892933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41521 (local address 127.15.56.2:41839)
0810 23:01:17.893458 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:17.893508 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:17.894579 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:17.895247 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:17.895259 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:17.895771 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:17.896348 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:17.896365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.897531 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.897535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:17.898650 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:17.898664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.899225 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.899230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:17.899409 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:17.900057 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:17.900076 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:17.901782 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:17.904130 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:17.904136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:17.904151 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:17.904520 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:17.904927 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:17.904931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:17.904933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:17.905109 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:17.905508 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:17.905514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:17.905772 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 23:01:17.905956 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 23:01:17.906239 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":133,"threads_started":1}
I20260810 23:01:17.908798 18958 heartbeater.cc:347] Connected to a master server at 127.15.56.62:41521
I20260810 23:01:17.908981 18958 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:17.909670 18958 heartbeater.cc:511] Master 127.15.56.62:41521 requested a full tablet report, sending...
I20260810 23:01:17.910414 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 18720
I20260810 23:01:17.910527 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:17.910806 15584 external_mini_cluster.cc:1428] Setting key 72ea1ad3666d380026e525bed7c17fb5
I20260810 23:01:17.910987 18372 ts_manager.cc:194] Registered new tserver with Master: f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973)
I20260810 23:01:17.911792 18372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:41839
I20260810 23:01:17.918692 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:17.911403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41521 (local address 127.15.56.2:37895)
0810 23:01:17.911717 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:17.911733 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:17.911846 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:17.912199 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:17.912202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:17.912459 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:17.912585 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:17.912592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.913364 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.913368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:17.913945 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:17.913953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.914406 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.914410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:17.914514 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:17.915044 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:17.915058 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:17.915398 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:17.917152 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:17.917155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:17.917156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:17.917369 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:17.917678 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:17.917681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:17.917683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:17.917765 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:17.918252 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:17.918254 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:17.918337 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 23:01:17.918425 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:01:17.918546 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":113,"threads_started":1}
I20260810 23:01:17.918813 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:17.911387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:37895 (local address 127.15.56.62:41521)
0810 23:01:17.911507 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:17.911511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:17.911813 (+ 302us) server_negotiation.cc:407] Connection header received
0810 23:01:17.911961 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:17.911964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:17.912005 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:17.912054 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:17.912699 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.913230 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.914055 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:17.914273 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:17.914371 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:17.915540 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:17.915544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:17.915547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:17.915573 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:17.917027 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:17.917473 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:17.917477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:17.917479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:17.917583 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:17.917849 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:17.917852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:17.917854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:17.918164 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:17.918271 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:17.918511 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 23:01:17.918641 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:01:17.940261 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41521
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:18.057847 18975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:18.058130 18975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:18.058187 18975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:18.062247 18975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:18.062309 18975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:18.062348 18975 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:01:18.062361 18975 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:01:18.062391 18975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:18.067821 18975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.56.62:41521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:18.068884 18975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:18.069896 18975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:18.079824 18982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:18.079926 18975 server_base.cc:1034] running on GCE node
W20260810 23:01:18.079824 18983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:18.079830 18985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:18.080494 18975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:18.081158 18975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:18.082337 18975 hybrid_clock.cc:648] HybridClock initialized: now 1786402878082317 us; error 33 us; skew 500 ppm
Aug 10 23:01:18 dist-test-slave-6bbx krb5kdc[18296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402878, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:18.088898 18975 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:01:18.089919 18975 webserver.cc:460] Webserver started at http://127.15.56.3:38157/ using document root <none> and password file <none>
I20260810 23:01:18.090540 18975 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:18.090612 18975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:18.090850 18975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:18.093083 18975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "b7e8a19b8df54298bfb197f611be5134"
format_stamp: "Formatted at 2026-08-10 23:01:18 on dist-test-slave-6bbx"
server_key: "8353c7391eba1c24673d3431bcc8bca0"
server_key_iv: "bf541395b4d59e77fa6f77af74733a52"
server_key_version: "encryptionkey@0"
I20260810 23:01:18.093669 18975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "b7e8a19b8df54298bfb197f611be5134"
format_stamp: "Formatted at 2026-08-10 23:01:18 on dist-test-slave-6bbx"
server_key: "8353c7391eba1c24673d3431bcc8bca0"
server_key_iv: "bf541395b4d59e77fa6f77af74733a52"
server_key_version: "encryptionkey@0"
I20260810 23:01:18.097946 18975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:01:18.100669 18996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:18.102036 18975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:18.102183 18975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "b7e8a19b8df54298bfb197f611be5134"
format_stamp: "Formatted at 2026-08-10 23:01:18 on dist-test-slave-6bbx"
server_key: "8353c7391eba1c24673d3431bcc8bca0"
server_key_iv: "bf541395b4d59e77fa6f77af74733a52"
server_key_version: "encryptionkey@0"
I20260810 23:01:18.102298 18975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:18.119607 18975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:18.123152 18975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:18.123365 18975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:18.125029 18975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:18.125089 18975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:18.125154 18975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:18.125182 18975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:01:18 dist-test-slave-6bbx krb5kdc[18296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402878, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:18.136577 18975 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:36885
I20260810 23:01:18.136652 19191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:36885 every 8 connection(s)
I20260810 23:01:18.137816 18975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:18.139961 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.126076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:56487 (local address 127.15.56.62:41521)
0810 23:01:18.126250 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.126254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.127473 (+ 1219us) server_negotiation.cc:407] Connection header received
0810 23:01:18.128798 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.128802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.128871 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.128976 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:18.130367 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.130961 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.132324 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.132583 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.132687 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.135779 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:18.135783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:18.135785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:18.135815 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:18.137473 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:18.138178 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:18.138181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:18.138183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:18.138302 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:18.138707 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:18.138710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:18.138712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:18.138953 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:18.139062 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.139655 (+ 593us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.139830 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 23:01:18.140839 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.126695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41521 (local address 127.15.56.3:56487)
0810 23:01:18.127305 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:18.127359 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:18.128530 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:18.129195 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:18.129203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:18.129639 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:18.130179 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.130196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.131090 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.131094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.132197 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.132208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.132772 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.132776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.132945 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.133599 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:18.133618 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:18.135489 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:18.137584 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:18.137590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:18.137606 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:18.137975 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:18.138433 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:18.138436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:18.138438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:18.138612 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:18.139084 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:18.139090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:18.139373 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 23:01:18.139541 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 23:01:18.139883 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":144,"threads_started":1}
I20260810 23:01:18.142248 19192 heartbeater.cc:347] Connected to a master server at 127.15.56.62:41521
I20260810 23:01:18.142407 19192 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:18.142966 19192 heartbeater.cc:511] Master 127.15.56.62:41521 requested a full tablet report, sending...
I20260810 23:01:18.144095 18372 ts_manager.cc:194] Registered new tserver with Master: b7e8a19b8df54298bfb197f611be5134 (127.15.56.3:36885)
I20260810 23:01:18.144771 18372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:56487
I20260810 23:01:18.147820 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 18975
I20260810 23:01:18.147950 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:18.148262 15584 external_mini_cluster.cc:1428] Setting key a979ed133490360e4d171e1b96e2968a
I20260810 23:01:18.150409 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:01:18.152127 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.144522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41521 (local address 127.15.56.3:39209)
0810 23:01:18.144829 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:18.144845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:18.144950 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:18.145227 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:18.145230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:18.145465 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:18.145574 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.145581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.146334 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.146337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.146906 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.146914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.147327 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.147331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.147423 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.148032 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:18.148045 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:18.148415 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:18.150359 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:18.150363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:18.150365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:18.150614 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:18.150936 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:18.150940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:18.150942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:18.151045 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:18.151526 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:18.151529 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:18.151632 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 23:01:18.151794 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 23:01:18.151958 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260810 23:01:18.152150 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.144610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:39209 (local address 127.15.56.62:41521)
0810 23:01:18.144751 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.144755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.144989 (+ 234us) server_negotiation.cc:407] Connection header received
0810 23:01:18.145039 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.145042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.145092 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.145200 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:18.145690 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.146219 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.147015 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.147226 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.147321 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.148548 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:18.148551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:18.148553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:18.148576 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:18.150257 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:18.150706 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:18.150710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:18.150712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:18.150809 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:18.151142 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:18.151146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:18.151147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:18.151384 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:18.151486 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.151859 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.151986 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 23:01:18 dist-test-slave-6bbx krb5kdc[18296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:01:18 dist-test-slave-6bbx krb5kdc[18296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:18.179212 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.170587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.15.56.62:41521)
0810 23:01:18.170756 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.170760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.170774 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:01:18.170818 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.170821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.170863 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.170939 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:18.171658 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.172274 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.173162 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.173420 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.173514 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.176203 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:18.176206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:18.176208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:18.176237 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:18.177438 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:18.177889 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:18.177892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:18.177893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:18.177982 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:18.178303 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:18.178306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:18.178307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:18.178508 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:18.178620 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.178914 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.179027 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:01:18.182746 18372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:01:18.185935 18372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:18.200841 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.194839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.15.56.1:45281)
0810 23:01:18.196090 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.196098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.196123 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:01:18.196206 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.196212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.196431 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.196611 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:18.196978 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.197832 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.199355 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.200135 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.200250 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.200398 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.200583 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.200697 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":76,"threads_started":1}
I20260810 23:01:18.200822 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.194716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:45281 (local address 127.0.0.1:51308)
0810 23:01:18.195123 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:18.195171 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:18.195299 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:18.196597 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:18.196604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:18.196629 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:18.196838 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.196846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.197944 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.197947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.199223 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.199235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.200271 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.200274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.200382 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.200424 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 23:01:18.200537 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 23:01:18.200657 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":48,"spinlock_wait_cycles":10880,"thread_start_us":84,"threads_started":1}
I20260810 23:01:18.200963 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.195522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.15.56.3:36885)
0810 23:01:18.196239 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.196249 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.196269 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:01:18.196331 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.196335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.196475 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.196846 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:18.197016 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.198185 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.199699 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.200444 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.200541 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.200693 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.200755 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.200836 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":104,"threads_started":1}
I20260810 23:01:18.200935 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.194747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:36885 (local address 127.0.0.1:47150)
0810 23:01:18.195429 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:18.195443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:18.195529 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:18.196620 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:18.196624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:18.196642 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:18.196876 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.196886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.198302 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.198306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.199554 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.199563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.200544 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.200547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.200640 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.200660 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 23:01:18.200711 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 23:01:18.200775 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":49,"spinlock_wait_cycles":11520,"thread_start_us":109,"threads_started":1}
I20260810 23:01:18.201591 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.195369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:34973 (local address 127.0.0.1:35882)
0810 23:01:18.195729 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:18.195742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:18.195823 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:18.196657 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:18.196660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:18.196686 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:18.197026 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.197032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.198363 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.198366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.199365 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.199376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.201175 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.201179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.201293 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.201318 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:01:18.201391 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 23:01:18.201457 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":201,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260810 23:01:18.202313 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.195760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.15.56.2:34973)
0810 23:01:18.196132 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.196159 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.196222 (+ 63us) server_negotiation.cc:407] Connection header received
0810 23:01:18.196308 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.196314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.196509 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.196836 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:18.197175 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.198271 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.199491 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.200141 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.200242 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.200415 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.202097 (+ 1682us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.202195 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":112,"threads_started":1}
I20260810 23:01:18.203884 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 9a300ed0c913430b8eb33dfa3b30a9bb (DEFAULT_TABLE table=test-table [id=315fdf1670cf4bd1ae0c207d443f8ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:18.204958 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a300ed0c913430b8eb33dfa3b30a9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:18.205128 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 9a300ed0c913430b8eb33dfa3b30a9bb (DEFAULT_TABLE table=test-table [id=315fdf1670cf4bd1ae0c207d443f8ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:18.205128 18836 tablet_service.cc:1467] Processing CreateTablet for tablet 9a300ed0c913430b8eb33dfa3b30a9bb (DEFAULT_TABLE table=test-table [id=315fdf1670cf4bd1ae0c207d443f8ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:18.206404 18836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a300ed0c913430b8eb33dfa3b30a9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:18.206411 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a300ed0c913430b8eb33dfa3b30a9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:18.213909 19224 tablet_bootstrap.cc:492] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: Bootstrap starting.
I20260810 23:01:18.215449 19226 tablet_bootstrap.cc:492] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: Bootstrap starting.
I20260810 23:01:18.215617 19227 tablet_bootstrap.cc:492] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: Bootstrap starting.
I20260810 23:01:18.217089 19224 tablet_bootstrap.cc:654] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:18.217753 19227 tablet_bootstrap.cc:654] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:18.217846 19226 tablet_bootstrap.cc:654] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:18.218099 19224 log.cc:824] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:18.218518 19227 log.cc:824] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:18.218695 19226 log.cc:824] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:18.220537 19224 tablet_bootstrap.cc:492] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: No bootstrap required, opened a new log
I20260810 23:01:18.220677 19227 tablet_bootstrap.cc:492] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: No bootstrap required, opened a new log
I20260810 23:01:18.220677 19226 tablet_bootstrap.cc:492] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: No bootstrap required, opened a new log
I20260810 23:01:18.220762 19224 ts_tablet_manager.cc:1397] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 23:01:18.220883 19227 ts_tablet_manager.cc:1397] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:01:18.220883 19226 ts_tablet_manager.cc:1397] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:01:18.224689 19224 raft_consensus.cc:348] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } }
I20260810 23:01:18.224942 19224 raft_consensus.cc:374] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:18.225018 19224 raft_consensus.cc:729] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4a4476bbd3417190567a6c72e00c5a, State: Initialized, Role: FOLLOWER
I20260810 23:01:18.224854 19226 raft_consensus.cc:348] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } }
I20260810 23:01:18.225142 19226 raft_consensus.cc:374] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:18.225236 19226 raft_consensus.cc:729] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8adb8adeb1f43b992722da92bcce13d, State: Initialized, Role: FOLLOWER
I20260810 23:01:18.225347 19227 raft_consensus.cc:348] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } }
I20260810 23:01:18.225509 19224 consensus_queue.cc:260] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } }
I20260810 23:01:18.225628 19227 raft_consensus.cc:374] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:18.225700 19227 raft_consensus.cc:729] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e8a19b8df54298bfb197f611be5134, State: Initialized, Role: FOLLOWER
I20260810 23:01:18.225766 19226 consensus_queue.cc:260] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } }
I20260810 23:01:18.226228 18702 heartbeater.cc:503] Master 127.15.56.62:41521 was elected leader, sending a full tablet report...
I20260810 23:01:18.226191 19227 consensus_queue.cc:260] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } }
I20260810 23:01:18.226512 18958 heartbeater.cc:503] Master 127.15.56.62:41521 was elected leader, sending a full tablet report...
I20260810 23:01:18.227075 19226 ts_tablet_manager.cc:1428] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:01:18.227075 19227 ts_tablet_manager.cc:1428] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:01:18.227433 19192 heartbeater.cc:503] Master 127.15.56.62:41521 was elected leader, sending a full tablet report...
I20260810 23:01:18.230048 19224 ts_tablet_manager.cc:1428] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260810 23:01:18.347230 19239 raft_consensus.cc:482] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:18.347473 19239 raft_consensus.cc:504] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } }
I20260810 23:01:18.348712 19239 leader_election.cc:288] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281), f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973)
I20260810 23:01:18.352795 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.348969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:45281 (local address 127.15.56.3:39477)
0810 23:01:18.349087 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:18.349103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:18.349193 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:18.349497 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:18.349500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:18.349519 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:18.349654 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.349660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.350488 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.350492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.351631 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.351643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.352406 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.352409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.352494 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.352516 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:01:18.352587 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 23:01:18.352655 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260810 23:01:18.352849 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.349078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:39477 (local address 127.15.56.1:45281)
0810 23:01:18.349259 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.349264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.349280 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:01:18.349339 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.349342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.349395 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.349473 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:18.349776 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.350332 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.351792 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.352287 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.352395 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.352465 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.352612 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.352709 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 23:01:18.353452 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.349178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:48089 (local address 127.15.56.2:34973)
0810 23:01:18.349319 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.349324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.349568 (+ 244us) server_negotiation.cc:407] Connection header received
0810 23:01:18.349663 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.349666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.349712 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.349775 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:18.350392 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.350939 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.352182 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.352794 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.352898 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.352959 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.353268 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.353344 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:01:18.353480 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.348980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:34973 (local address 127.15.56.3:48089)
0810 23:01:18.349477 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:18.349488 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:18.349579 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:18.349800 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:18.349804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:18.349819 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:18.350272 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.350280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.351053 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.351057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.352062 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.352071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.353055 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.353059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.353152 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.353175 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:01:18.353248 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 23:01:18.353319 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":139,"thread_start_us":163,"threads_started":1}
I20260810 23:01:18.353499 18626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a300ed0c913430b8eb33dfa3b30a9bb" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4476bbd3417190567a6c72e00c5a" is_pre_election: true
I20260810 23:01:18.353884 18626 raft_consensus.cc:2454] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 0.
I20260810 23:01:18.354468 18997 leader_election.cc:302] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7e8a19b8df54298bfb197f611be5134, df4a4476bbd3417190567a6c72e00c5a; no voters:
I20260810 23:01:18.354609 18868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a300ed0c913430b8eb33dfa3b30a9bb" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8adb8adeb1f43b992722da92bcce13d" is_pre_election: true
I20260810 23:01:18.354780 19239 raft_consensus.cc:2781] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:18.354884 19239 raft_consensus.cc:482] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:18.354934 19239 raft_consensus.cc:3037] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:18.354943 18868 raft_consensus.cc:2454] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 0.
I20260810 23:01:18.356215 19239 raft_consensus.cc:504] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } }
I20260810 23:01:18.356547 19239 leader_election.cc:288] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 election: Requested vote from peers df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281), f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973)
I20260810 23:01:18.356915 18868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a300ed0c913430b8eb33dfa3b30a9bb" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8adb8adeb1f43b992722da92bcce13d"
I20260810 23:01:18.357025 18868 raft_consensus.cc:3037] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:18.358421 18868 raft_consensus.cc:2454] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 1.
I20260810 23:01:18.358737 18998 leader_election.cc:302] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7e8a19b8df54298bfb197f611be5134, f8adb8adeb1f43b992722da92bcce13d; no voters:
I20260810 23:01:18.358721 18626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a300ed0c913430b8eb33dfa3b30a9bb" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4476bbd3417190567a6c72e00c5a"
I20260810 23:01:18.358855 18626 raft_consensus.cc:3037] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:18.358942 19239 raft_consensus.cc:2781] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:18.359165 19239 raft_consensus.cc:686] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 1 LEADER]: Becoming Leader. State: Replica: b7e8a19b8df54298bfb197f611be5134, State: Running, Role: LEADER
I20260810 23:01:18.359511 19239 consensus_queue.cc:237] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } }
I20260810 23:01:18.360075 18626 raft_consensus.cc:2454] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 1.
I20260810 23:01:18.362915 18372 catalog_manager.cc:5495] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7e8a19b8df54298bfb197f611be5134 (127.15.56.3). New cstate: current_term: 1 leader_uuid: "b7e8a19b8df54298bfb197f611be5134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } health_report { overall_health: UNKNOWN } } }
W20260810 23:01:18.388937 19193 tablet.cc:2369] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:01:18.405267 18959 tablet.cc:2369] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:01:18.430306 18704 tablet.cc:2369] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 23:01:18 dist-test-slave-6bbx krb5kdc[18296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402877, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:18.434240 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.425611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41521 (local address 127.0.0.1:45244)
0810 23:01:18.425977 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:18.425990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:18.426104 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:18.426416 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:18.426419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:18.426880 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:18.426985 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.426992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.427861 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.427864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.428450 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.428458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.428933 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.428936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.429008 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.429593 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:18.429605 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:18.431196 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:18.432708 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:18.432717 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:18.432723 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:18.432963 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:18.433269 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:18.433272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:18.433273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:18.433355 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:18.433762 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:18.433768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:18.433875 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 23:01:18.433965 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:01:18.434084 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":136,"threads_started":1}
I20260810 23:01:18.434279 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.425701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.15.56.62:41521)
0810 23:01:18.425842 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.425846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.426067 (+ 221us) server_negotiation.cc:407] Connection header received
0810 23:01:18.426221 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.426223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.426270 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.426346 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:18.427151 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.427729 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.428584 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.428809 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.428899 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.431335 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:18.431338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:18.431340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:18.431361 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:18.432589 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:18.433080 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:18.433083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:18.433084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:18.433182 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:18.433469 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:18.433472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:18.433474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:18.433674 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:18.433760 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.434035 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.434168 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:01:18.438429 18372 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45244:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:01:18.439183 18372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:18.444490 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 47c147bb001e493c9d202456f5dfe990 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5fa7b56bec14fc9aae6fce12c313bb5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:01:18.444335 18836 tablet_service.cc:1467] Processing CreateTablet for tablet 47c147bb001e493c9d202456f5dfe990 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5fa7b56bec14fc9aae6fce12c313bb5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:01:18.444350 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 47c147bb001e493c9d202456f5dfe990 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5fa7b56bec14fc9aae6fce12c313bb5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:01:18.445122 18836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c147bb001e493c9d202456f5dfe990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:18.445130 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c147bb001e493c9d202456f5dfe990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:18.445122 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c147bb001e493c9d202456f5dfe990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:18.447799 19227 tablet_bootstrap.cc:492] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: Bootstrap starting.
I20260810 23:01:18.448110 19224 tablet_bootstrap.cc:492] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a: Bootstrap starting.
I20260810 23:01:18.448248 19226 tablet_bootstrap.cc:492] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d: Bootstrap starting.
I20260810 23:01:18.448863 19227 tablet_bootstrap.cc:654] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:18.449028 19226 tablet_bootstrap.cc:654] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:18.449061 19224 tablet_bootstrap.cc:654] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:18.450197 19226 tablet_bootstrap.cc:492] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d: No bootstrap required, opened a new log
I20260810 23:01:18.450281 19226 ts_tablet_manager.cc:1397] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:01:18.450431 19224 tablet_bootstrap.cc:492] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a: No bootstrap required, opened a new log
I20260810 23:01:18.450446 19227 tablet_bootstrap.cc:492] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: No bootstrap required, opened a new log
I20260810 23:01:18.450511 19227 ts_tablet_manager.cc:1397] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:18.450511 19224 ts_tablet_manager.cc:1397] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:18.450675 19226 raft_consensus.cc:348] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.450762 19226 raft_consensus.cc:374] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:18.450783 19226 raft_consensus.cc:729] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8adb8adeb1f43b992722da92bcce13d, State: Initialized, Role: FOLLOWER
I20260810 23:01:18.450888 19226 consensus_queue.cc:260] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.450944 19227 raft_consensus.cc:348] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.450944 19224 raft_consensus.cc:348] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.451051 19227 raft_consensus.cc:374] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:18.451059 19224 raft_consensus.cc:374] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:18.451083 19227 raft_consensus.cc:729] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e8a19b8df54298bfb197f611be5134, State: Initialized, Role: FOLLOWER
I20260810 23:01:18.451090 19224 raft_consensus.cc:729] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4a4476bbd3417190567a6c72e00c5a, State: Initialized, Role: FOLLOWER
I20260810 23:01:18.451201 19227 consensus_queue.cc:260] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.451211 19224 consensus_queue.cc:260] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.451296 19226 ts_tablet_manager.cc:1428] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:18.451462 19224 ts_tablet_manager.cc:1428] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:18.451462 19227 ts_tablet_manager.cc:1428] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:18.451537 19239 tablet_replica.cc:439] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } }
I20260810 23:01:18.451720 19239 tablet_replica.cc:442] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:18.451784 19234 tablet_replica.cc:439] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } }
I20260810 23:01:18.451912 19234 tablet_replica.cc:442] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:18.452472 19235 tablet_replica.cc:439] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } }
I20260810 23:01:18.452605 19235 tablet_replica.cc:442] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:18.656565 19239 raft_consensus.cc:482] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:18.656718 19239 raft_consensus.cc:504] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.657151 19239 leader_election.cc:288] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281), f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973)
I20260810 23:01:18.657696 18626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c147bb001e493c9d202456f5dfe990" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4476bbd3417190567a6c72e00c5a" is_pre_election: true
I20260810 23:01:18.657694 18868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c147bb001e493c9d202456f5dfe990" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8adb8adeb1f43b992722da92bcce13d" is_pre_election: true
I20260810 23:01:18.657840 18868 raft_consensus.cc:2454] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 0.
I20260810 23:01:18.657847 18626 raft_consensus.cc:2454] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 0.
I20260810 23:01:18.658169 18997 leader_election.cc:302] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b7e8a19b8df54298bfb197f611be5134, df4a4476bbd3417190567a6c72e00c5a, f8adb8adeb1f43b992722da92bcce13d; no voters:
I20260810 23:01:18.658350 19239 raft_consensus.cc:2781] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:18.658414 19239 raft_consensus.cc:482] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:18.658461 19239 raft_consensus.cc:3037] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:18.659489 19239 raft_consensus.cc:504] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.659879 19239 leader_election.cc:288] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 election: Requested vote from peers df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281), f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973)
I20260810 23:01:18.660269 18626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c147bb001e493c9d202456f5dfe990" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4476bbd3417190567a6c72e00c5a"
I20260810 23:01:18.660276 18868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c147bb001e493c9d202456f5dfe990" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8adb8adeb1f43b992722da92bcce13d"
I20260810 23:01:18.660377 18868 raft_consensus.cc:3037] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:18.660386 18626 raft_consensus.cc:3037] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:18.661306 18868 raft_consensus.cc:2454] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 1.
I20260810 23:01:18.661326 18626 raft_consensus.cc:2454] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 1.
I20260810 23:01:18.661594 18997 leader_election.cc:302] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b7e8a19b8df54298bfb197f611be5134, df4a4476bbd3417190567a6c72e00c5a, f8adb8adeb1f43b992722da92bcce13d; no voters:
I20260810 23:01:18.661801 19239 raft_consensus.cc:2781] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:18.661892 19239 raft_consensus.cc:686] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [term 1 LEADER]: Becoming Leader. State: Replica: b7e8a19b8df54298bfb197f611be5134, State: Running, Role: LEADER
I20260810 23:01:18.662034 19239 consensus_queue.cc:237] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.662688 19243 tablet_replica.cc:439] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: TxnStatusTablet state changed. Reason: New leader b7e8a19b8df54298bfb197f611be5134. Latest consensus state: current_term: 1 leader_uuid: "b7e8a19b8df54298bfb197f611be5134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } }
I20260810 23:01:18.662806 19243 tablet_replica.cc:442] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:01:18.663060 19260 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:01:18.663341 19260 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:01:18.663882 18372 catalog_manager.cc:5495] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7e8a19b8df54298bfb197f611be5134 (127.15.56.3). New cstate: current_term: 1 leader_uuid: "b7e8a19b8df54298bfb197f611be5134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260810 23:01:18.695709 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.692020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:36885 (local address 127.0.0.1:47166)
0810 23:01:18.692261 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:18.692276 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:18.692388 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:18.692701 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:18.692704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:18.692711 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:01:18.692818 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.692823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.693528 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.693532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.694142 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.694150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.694572 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.694575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.694666 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.694719 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:01:18.695373 (+ 654us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:01:18.695390 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:01:18.695457 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 23:01:18.695547 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260810 23:01:18.695700 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.692135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.15.56.3:36885)
0810 23:01:18.692271 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.692276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.692321 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:01:18.692526 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.692529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.692590 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.692670 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:18.692917 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.693427 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.694254 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.694475 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.694559 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.694858 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:18.695280 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:18.695385 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.695467 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.695528 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:01:18.701182 18626 raft_consensus.cc:1261] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Refusing update from remote peer b7e8a19b8df54298bfb197f611be5134: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:18.701182 18868 raft_consensus.cc:1261] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Refusing update from remote peer b7e8a19b8df54298bfb197f611be5134: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:18.701915 19239 consensus_queue.cc:1035] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:18.702109 19243 consensus_queue.cc:1035] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:18.706346 19235 tablet_replica.cc:439] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d: TxnStatusTablet state changed. Reason: New leader b7e8a19b8df54298bfb197f611be5134. Latest consensus state: current_term: 1 leader_uuid: "b7e8a19b8df54298bfb197f611be5134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } }
I20260810 23:01:18.706473 19235 tablet_replica.cc:442] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:18.706881 19234 tablet_replica.cc:439] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a: TxnStatusTablet state changed. Reason: New leader b7e8a19b8df54298bfb197f611be5134. Latest consensus state: current_term: 1 leader_uuid: "b7e8a19b8df54298bfb197f611be5134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } }
I20260810 23:01:18.706993 19234 tablet_replica.cc:442] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:18.710554 19243 tablet_replica.cc:439] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7e8a19b8df54298bfb197f611be5134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } }
I20260810 23:01:18.710691 19243 tablet_replica.cc:442] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:01:18.710842 19243 tablet_replica.cc:439] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7e8a19b8df54298bfb197f611be5134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } }
I20260810 23:01:18.710906 19243 tablet_replica.cc:442] T 47c147bb001e493c9d202456f5dfe990 P b7e8a19b8df54298bfb197f611be5134: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:01:18.711229 19264 mvcc.cc:205] Tried to move back new op lower bound from 7317106191152709632 to 7317106191001206784. Current Snapshot: MvccSnapshot[applied={T|T < 7317106191152709632}]
I20260810 23:01:18.711403 19234 tablet_replica.cc:439] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7e8a19b8df54298bfb197f611be5134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } }
I20260810 23:01:18.711504 19234 tablet_replica.cc:442] T 47c147bb001e493c9d202456f5dfe990 P df4a4476bbd3417190567a6c72e00c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:18.712159 19235 tablet_replica.cc:439] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7e8a19b8df54298bfb197f611be5134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } } }
I20260810 23:01:18.712278 19235 tablet_replica.cc:442] T 47c147bb001e493c9d202456f5dfe990 P f8adb8adeb1f43b992722da92bcce13d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:18.712360 19266 mvcc.cc:205] Tried to move back new op lower bound from 7317106191152709632 to 7317106191001206784. Current Snapshot: MvccSnapshot[applied={T|T < 7317106191152709632}]
I20260810 23:01:18.712553 19265 mvcc.cc:205] Tried to move back new op lower bound from 7317106191152709632 to 7317106191001206784. Current Snapshot: MvccSnapshot[applied={T|T < 7317106191152709632}]
I20260810 23:01:18.724248 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.721005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.15.56.3:36885)
0810 23:01:18.721114 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.721119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.721292 (+ 173us) server_negotiation.cc:407] Connection header received
0810 23:01:18.721426 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.721429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.721477 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.721536 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:18.721798 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.722326 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.723190 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.723435 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.723521 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.723801 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:18.723874 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:18.723937 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.724029 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.724082 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 23:01:18.736654 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.733183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:40203 (local address 127.15.56.3:36885)
0810 23:01:18.733296 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:18.733301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:18.733505 (+ 204us) server_negotiation.cc:407] Connection header received
0810 23:01:18.733620 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:18.733623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:18.733663 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:18.733723 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:18.733974 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.734492 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.735374 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.735758 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.735869 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.736202 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:18.736292 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:18.736363 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:18.736452 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 23:01:18.736507 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":38}
I20260810 23:01:18.736667 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:18.733117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:36885 (local address 127.15.56.3:40203)
0810 23:01:18.733423 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:18.733434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:18.733544 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:18.733735 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:18.733738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:18.733745 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:01:18.733881 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.733888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.734592 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.734596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.735245 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:18.735255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:18.735897 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:18.735900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:18.735988 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:18.736042 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:01:18.736375 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:01:18.736379 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:01:18.736422 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 23:01:18.736490 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":17,"thread_start_us":122,"threads_started":1}
I20260810 23:01:18.744778 18868 raft_consensus.cc:1261] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Refusing update from remote peer b7e8a19b8df54298bfb197f611be5134: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:18.744778 18626 raft_consensus.cc:1261] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Refusing update from remote peer b7e8a19b8df54298bfb197f611be5134: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:18.745198 19272 consensus_queue.cc:1035] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:18.745388 19243 consensus_queue.cc:1035] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:18.789970 18370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:18.790191 18370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:18.793051 18370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850: Sending DeleteTablet for 3 replicas of tablet 9a300ed0c913430b8eb33dfa3b30a9bb
I20260810 23:01:18.793828 19077 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a300ed0c913430b8eb33dfa3b30a9bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:18 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:47150
I20260810 23:01:18.793833 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a300ed0c913430b8eb33dfa3b30a9bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:18 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:51308
I20260810 23:01:18.794147 18836 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a300ed0c913430b8eb33dfa3b30a9bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:18 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:35882
I20260810 23:01:18.794425 19292 tablet_replica.cc:330] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: stopping tablet replica
I20260810 23:01:18.794631 19296 tablet_replica.cc:330] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: stopping tablet replica
I20260810 23:01:18.794822 19292 raft_consensus.cc:2229] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:18.795042 19296 raft_consensus.cc:2229] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:18.795109 19292 raft_consensus.cc:2258] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:18.795347 19296 raft_consensus.cc:2258] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:18.795365 19290 tablet_replica.cc:330] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: stopping tablet replica
I20260810 23:01:18.795362 18370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:01:18.795634 19290 raft_consensus.cc:2229] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 1 LEADER]: Raft consensus shutting down.
W20260810 23:01:18.795812 18370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:18.796022 19290 raft_consensus.cc:2258] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:18.796941 19296 ts_tablet_manager.cc:1905] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:18.797243 19290 ts_tablet_manager.cc:1905] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:18.800246 19290 ts_tablet_manager.cc:1918] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:18.800252 19296 ts_tablet_manager.cc:1918] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:18.800364 19296 log.cc:1196] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/9a300ed0c913430b8eb33dfa3b30a9bb
I20260810 23:01:18.800434 19290 log.cc:1196] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/9a300ed0c913430b8eb33dfa3b30a9bb
I20260810 23:01:18.800812 19296 ts_tablet_manager.cc:1939] T 9a300ed0c913430b8eb33dfa3b30a9bb P f8adb8adeb1f43b992722da92bcce13d: Deleting consensus metadata
I20260810 23:01:18.800896 19290 ts_tablet_manager.cc:1939] T 9a300ed0c913430b8eb33dfa3b30a9bb P b7e8a19b8df54298bfb197f611be5134: Deleting consensus metadata
I20260810 23:01:18.802160 18347 catalog_manager.cc:4841] TS b7e8a19b8df54298bfb197f611be5134 (127.15.56.3:36885): tablet 9a300ed0c913430b8eb33dfa3b30a9bb (table test-table [id=315fdf1670cf4bd1ae0c207d443f8ec3]) successfully deleted
I20260810 23:01:18.802255 19077 tablet_service.cc:1467] Processing CreateTablet for tablet aaa4ee83aeb048f1bb05efc2db560773 (DEFAULT_TABLE table=test-table [id=9f7d075f737f4c1caf84ff316949523e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:18.802512 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa4ee83aeb048f1bb05efc2db560773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:18.804283 18345 catalog_manager.cc:4841] TS f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973): tablet 9a300ed0c913430b8eb33dfa3b30a9bb (table test-table [id=315fdf1670cf4bd1ae0c207d443f8ec3]) successfully deleted
I20260810 23:01:18.804466 19227 tablet_bootstrap.cc:492] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134: Bootstrap starting.
I20260810 23:01:18.804622 18836 tablet_service.cc:1467] Processing CreateTablet for tablet aaa4ee83aeb048f1bb05efc2db560773 (DEFAULT_TABLE table=test-table [id=9f7d075f737f4c1caf84ff316949523e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:18.804865 18836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa4ee83aeb048f1bb05efc2db560773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:18.803974 18590 tablet_service.cc:1467] Processing CreateTablet for tablet aaa4ee83aeb048f1bb05efc2db560773 (DEFAULT_TABLE table=test-table [id=9f7d075f737f4c1caf84ff316949523e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:18.805130 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaa4ee83aeb048f1bb05efc2db560773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:18.805389 19227 tablet_bootstrap.cc:654] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:18.806856 19227 tablet_bootstrap.cc:492] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134: No bootstrap required, opened a new log
I20260810 23:01:18.806951 19227 ts_tablet_manager.cc:1397] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:18.807374 19226 tablet_bootstrap.cc:492] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d: Bootstrap starting.
I20260810 23:01:18.807428 19227 raft_consensus.cc:348] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.807546 19227 raft_consensus.cc:374] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:18.807578 19227 raft_consensus.cc:729] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e8a19b8df54298bfb197f611be5134, State: Initialized, Role: FOLLOWER
I20260810 23:01:18.807751 19224 tablet_bootstrap.cc:492] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a: Bootstrap starting.
I20260810 23:01:18.807737 19227 consensus_queue.cc:260] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.808015 19227 ts_tablet_manager.cc:1428] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:01:18.808382 19226 tablet_bootstrap.cc:654] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:18.808624 19224 tablet_bootstrap.cc:654] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:18.809850 19224 tablet_bootstrap.cc:492] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a: No bootstrap required, opened a new log
I20260810 23:01:18.809938 19224 ts_tablet_manager.cc:1397] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:01:18.810161 19226 tablet_bootstrap.cc:492] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d: No bootstrap required, opened a new log
I20260810 23:01:18.810243 19226 ts_tablet_manager.cc:1397] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:18.810385 19224 raft_consensus.cc:348] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.810496 19224 raft_consensus.cc:374] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:18.810526 19224 raft_consensus.cc:729] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4a4476bbd3417190567a6c72e00c5a, State: Initialized, Role: FOLLOWER
I20260810 23:01:18.810647 19224 consensus_queue.cc:260] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.810709 19226 raft_consensus.cc:348] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.810827 19226 raft_consensus.cc:374] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:18.810863 19226 raft_consensus.cc:729] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8adb8adeb1f43b992722da92bcce13d, State: Initialized, Role: FOLLOWER
I20260810 23:01:18.810912 19224 ts_tablet_manager.cc:1428] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:01:18.810985 19226 consensus_queue.cc:260] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.811256 19226 ts_tablet_manager.cc:1428] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:18.812837 19292 ts_tablet_manager.cc:1905] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:18.816200 19292 ts_tablet_manager.cc:1918] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:18.816310 19292 log.cc:1196] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/wal/wals/9a300ed0c913430b8eb33dfa3b30a9bb
I20260810 23:01:18.816586 19292 ts_tablet_manager.cc:1939] T 9a300ed0c913430b8eb33dfa3b30a9bb P df4a4476bbd3417190567a6c72e00c5a: Deleting consensus metadata
I20260810 23:01:18.817757 18343 catalog_manager.cc:4841] TS df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281): tablet 9a300ed0c913430b8eb33dfa3b30a9bb (table test-table [id=315fdf1670cf4bd1ae0c207d443f8ec3]) successfully deleted
I20260810 23:01:18.966722 19243 raft_consensus.cc:482] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:18.966857 19243 raft_consensus.cc:504] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.967211 19243 leader_election.cc:288] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281), f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973)
I20260810 23:01:18.967703 18625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa4ee83aeb048f1bb05efc2db560773" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4476bbd3417190567a6c72e00c5a" is_pre_election: true
I20260810 23:01:18.967700 18868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa4ee83aeb048f1bb05efc2db560773" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8adb8adeb1f43b992722da92bcce13d" is_pre_election: true
I20260810 23:01:18.967873 18868 raft_consensus.cc:2454] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 0.
I20260810 23:01:18.967882 18625 raft_consensus.cc:2454] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 0.
I20260810 23:01:18.968201 18998 leader_election.cc:302] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b7e8a19b8df54298bfb197f611be5134, df4a4476bbd3417190567a6c72e00c5a, f8adb8adeb1f43b992722da92bcce13d; no voters:
I20260810 23:01:18.968395 19243 raft_consensus.cc:2781] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:18.968477 19243 raft_consensus.cc:482] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:18.968525 19243 raft_consensus.cc:3037] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:18.969632 19243 raft_consensus.cc:504] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.969946 19243 leader_election.cc:288] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 election: Requested vote from peers df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281), f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973)
I20260810 23:01:18.970345 18625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa4ee83aeb048f1bb05efc2db560773" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4476bbd3417190567a6c72e00c5a"
I20260810 23:01:18.970345 18868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaa4ee83aeb048f1bb05efc2db560773" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8adb8adeb1f43b992722da92bcce13d"
I20260810 23:01:18.970477 18625 raft_consensus.cc:3037] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:18.970476 18868 raft_consensus.cc:3037] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:18.971596 18625 raft_consensus.cc:2454] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 1.
I20260810 23:01:18.971596 18868 raft_consensus.cc:2454] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 1.
I20260810 23:01:18.971925 18998 leader_election.cc:302] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7e8a19b8df54298bfb197f611be5134, f8adb8adeb1f43b992722da92bcce13d; no voters:
I20260810 23:01:18.972122 19243 raft_consensus.cc:2781] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:18.972216 19243 raft_consensus.cc:686] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 1 LEADER]: Becoming Leader. State: Replica: b7e8a19b8df54298bfb197f611be5134, State: Running, Role: LEADER
I20260810 23:01:18.972349 19243 consensus_queue.cc:237] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:18.973939 18370 catalog_manager.cc:5495] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7e8a19b8df54298bfb197f611be5134 (127.15.56.3). New cstate: current_term: 1 leader_uuid: "b7e8a19b8df54298bfb197f611be5134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260810 23:01:19.017834 18625 raft_consensus.cc:1261] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Refusing update from remote peer b7e8a19b8df54298bfb197f611be5134: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:19.017874 18868 raft_consensus.cc:1261] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Refusing update from remote peer b7e8a19b8df54298bfb197f611be5134: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:19.018227 19239 consensus_queue.cc:1035] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:19.018392 19243 consensus_queue.cc:1035] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:01:19.031456 19264 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:01:20.582382 18368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:20.582547 18368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:20.585280 18368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850: Sending DeleteTablet for 3 replicas of tablet aaa4ee83aeb048f1bb05efc2db560773
I20260810 23:01:20.585927 19077 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa4ee83aeb048f1bb05efc2db560773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:20 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:47150
I20260810 23:01:20.585940 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa4ee83aeb048f1bb05efc2db560773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:20 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:51308
I20260810 23:01:20.586019 18836 tablet_service.cc:1514] Processing DeleteTablet for tablet aaa4ee83aeb048f1bb05efc2db560773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:20 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:35882
I20260810 23:01:20.586311 19335 tablet_replica.cc:330] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134: stopping tablet replica
I20260810 23:01:20.586383 19338 tablet_replica.cc:330] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d: stopping tablet replica
I20260810 23:01:20.586457 19335 raft_consensus.cc:2229] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:20.586546 19338 raft_consensus.cc:2229] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:20.586697 19335 raft_consensus.cc:2258] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:20.586731 19338 raft_consensus.cc:2258] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:20.586928 19336 tablet_replica.cc:330] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a: stopping tablet replica
I20260810 23:01:20.587095 19336 raft_consensus.cc:2229] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:20.587229 19336 raft_consensus.cc:2258] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:20.587524 19338 ts_tablet_manager.cc:1905] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:20.587971 19336 ts_tablet_manager.cc:1905] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:20.588442 19335 ts_tablet_manager.cc:1905] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:20.590346 19338 ts_tablet_manager.cc:1918] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:01:20.590432 19338 log.cc:1196] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/aaa4ee83aeb048f1bb05efc2db560773
I20260810 23:01:20.590766 19338 ts_tablet_manager.cc:1939] T aaa4ee83aeb048f1bb05efc2db560773 P f8adb8adeb1f43b992722da92bcce13d: Deleting consensus metadata
I20260810 23:01:20.591085 19336 ts_tablet_manager.cc:1918] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:01:20.591168 19336 log.cc:1196] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/wal/wals/aaa4ee83aeb048f1bb05efc2db560773
I20260810 23:01:20.591437 19336 ts_tablet_manager.cc:1939] T aaa4ee83aeb048f1bb05efc2db560773 P df4a4476bbd3417190567a6c72e00c5a: Deleting consensus metadata
I20260810 23:01:20.591655 18345 catalog_manager.cc:4841] TS f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973): tablet aaa4ee83aeb048f1bb05efc2db560773 (table test-table [id=9f7d075f737f4c1caf84ff316949523e]) successfully deleted
I20260810 23:01:20.592133 18343 catalog_manager.cc:4841] TS df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281): tablet aaa4ee83aeb048f1bb05efc2db560773 (table test-table [id=9f7d075f737f4c1caf84ff316949523e]) successfully deleted
I20260810 23:01:20.593865 19335 ts_tablet_manager.cc:1918] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:01:20.593935 19335 log.cc:1196] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/aaa4ee83aeb048f1bb05efc2db560773
I20260810 23:01:20.594193 19335 ts_tablet_manager.cc:1939] T aaa4ee83aeb048f1bb05efc2db560773 P b7e8a19b8df54298bfb197f611be5134: Deleting consensus metadata
I20260810 23:01:20.594857 18347 catalog_manager.cc:4841] TS b7e8a19b8df54298bfb197f611be5134 (127.15.56.3:36885): tablet aaa4ee83aeb048f1bb05efc2db560773 (table test-table [id=9f7d075f737f4c1caf84ff316949523e]) successfully deleted
Aug 10 23:01:20 dist-test-slave-6bbx krb5kdc[18296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402880, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:01:20 dist-test-slave-6bbx krb5kdc[18296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402880, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:20.615628 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:20.606420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.15.56.62:41521)
0810 23:01:20.606705 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:20.606719 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:20.606738 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:01:20.606843 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:20.606846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:20.606895 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:20.606987 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:20.607854 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:20.608517 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:20.609540 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:20.609777 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:20.609857 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:20.612329 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:20.612333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:20.612335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:20.612362 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:20.613927 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:20.614360 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:20.614364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:20.614365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:20.614453 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:20.614714 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:20.614718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:20.614720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:20.614921 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:20.615049 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:20.615293 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 23:01:20.615430 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260810 23:01:20.618422 18368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:01:20.618894 18368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:20.623958 18590 tablet_service.cc:1467] Processing CreateTablet for tablet c9c21c7e41d647f181a2a48637caae3f (DEFAULT_TABLE table=test-table [id=02004f0fb89945568ff37676beb02f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:20.624007 18836 tablet_service.cc:1467] Processing CreateTablet for tablet c9c21c7e41d647f181a2a48637caae3f (DEFAULT_TABLE table=test-table [id=02004f0fb89945568ff37676beb02f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:20.624100 19077 tablet_service.cc:1467] Processing CreateTablet for tablet c9c21c7e41d647f181a2a48637caae3f (DEFAULT_TABLE table=test-table [id=02004f0fb89945568ff37676beb02f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:20.624311 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c21c7e41d647f181a2a48637caae3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:20.624356 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c21c7e41d647f181a2a48637caae3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:20.624351 18836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c21c7e41d647f181a2a48637caae3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:20.626708 19349 tablet_bootstrap.cc:492] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d: Bootstrap starting.
I20260810 23:01:20.627007 19351 tablet_bootstrap.cc:492] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a: Bootstrap starting.
I20260810 23:01:20.627254 19350 tablet_bootstrap.cc:492] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134: Bootstrap starting.
I20260810 23:01:20.627764 19349 tablet_bootstrap.cc:654] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:20.628013 19351 tablet_bootstrap.cc:654] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:20.628136 19350 tablet_bootstrap.cc:654] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:20.629307 19351 tablet_bootstrap.cc:492] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a: No bootstrap required, opened a new log
I20260810 23:01:20.629328 19349 tablet_bootstrap.cc:492] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d: No bootstrap required, opened a new log
I20260810 23:01:20.629396 19351 ts_tablet_manager.cc:1397] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:20.629416 19349 ts_tablet_manager.cc:1397] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 23:01:20.629891 19351 raft_consensus.cc:348] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:20.629915 19349 raft_consensus.cc:348] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:20.630020 19351 raft_consensus.cc:374] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:20.630031 19349 raft_consensus.cc:374] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:20.630057 19351 raft_consensus.cc:729] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4a4476bbd3417190567a6c72e00c5a, State: Initialized, Role: FOLLOWER
I20260810 23:01:20.630077 19349 raft_consensus.cc:729] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8adb8adeb1f43b992722da92bcce13d, State: Initialized, Role: FOLLOWER
I20260810 23:01:20.630201 19351 consensus_queue.cc:260] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:20.630215 19349 consensus_queue.cc:260] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:20.630600 19351 ts_tablet_manager.cc:1428] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:20.630611 19349 ts_tablet_manager.cc:1428] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 23:01:20.631541 19350 tablet_bootstrap.cc:492] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134: No bootstrap required, opened a new log
I20260810 23:01:20.631637 19350 ts_tablet_manager.cc:1397] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260810 23:01:20.632076 19350 raft_consensus.cc:348] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:20.632202 19350 raft_consensus.cc:374] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:20.632246 19350 raft_consensus.cc:729] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e8a19b8df54298bfb197f611be5134, State: Initialized, Role: FOLLOWER
I20260810 23:01:20.632380 19350 consensus_queue.cc:260] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:20.632640 19350 ts_tablet_manager.cc:1428] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:20.972827 19356 raft_consensus.cc:482] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:20.973011 19356 raft_consensus.cc:504] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:20.974193 19356 leader_election.cc:288] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281), b7e8a19b8df54298bfb197f611be5134 (127.15.56.3:36885)
I20260810 23:01:20.978371 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:20.974589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:53317 (local address 127.15.56.3:36885)
0810 23:01:20.974897 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:20.974906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:20.974995 (+ 89us) server_negotiation.cc:407] Connection header received
0810 23:01:20.975168 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:20.975170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:20.975213 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:20.975292 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:20.975549 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:20.976200 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:20.977369 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:20.977782 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:20.977881 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:20.977951 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:20.978124 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 23:01:20.978201 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
I20260810 23:01:20.978389 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:20.974616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:45517 (local address 127.15.56.1:45281)
0810 23:01:20.974902 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:20.974911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:20.974997 (+ 86us) server_negotiation.cc:407] Connection header received
0810 23:01:20.975168 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:20.975171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:20.975218 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:20.975290 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:20.975557 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:20.976191 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:20.977375 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:20.977800 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:20.977895 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:20.977962 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:20.978146 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 23:01:20.978233 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
I20260810 23:01:20.978395 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:20.974489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:36885 (local address 127.15.56.2:53317)
0810 23:01:20.974897 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:20.974912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:20.975065 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:20.975305 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:20.975308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:20.975324 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:20.975448 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:20.975454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:20.976337 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:20.976340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:20.977242 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:20.977252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:20.977905 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:20.977908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:20.978005 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:20.978028 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:01:20.978094 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 23:01:20.978201 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":68,"spinlock_wait_cycles":9216,"thread_start_us":119,"threads_started":1}
I20260810 23:01:20.978399 19360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:20.974465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:45281 (local address 127.15.56.2:45517)
0810 23:01:20.974902 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:20.974915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:20.975064 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:20.975315 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:20.975318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:20.975334 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:20.975454 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:20.975459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:20.976308 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:20.976311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:20.977253 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:20.977263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:20.977936 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:20.977939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:20.978033 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:20.978050 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:01:20.978114 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 23:01:20.978206 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":30,"spinlock_wait_cycles":28160,"thread_start_us":158,"threads_started":1}
I20260810 23:01:20.978987 18625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c21c7e41d647f181a2a48637caae3f" candidate_uuid: "f8adb8adeb1f43b992722da92bcce13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4476bbd3417190567a6c72e00c5a" is_pre_election: true
I20260810 23:01:20.979032 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c21c7e41d647f181a2a48637caae3f" candidate_uuid: "f8adb8adeb1f43b992722da92bcce13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7e8a19b8df54298bfb197f611be5134" is_pre_election: true
I20260810 23:01:20.979143 18625 raft_consensus.cc:2454] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8adb8adeb1f43b992722da92bcce13d in term 0.
I20260810 23:01:20.979269 19115 raft_consensus.cc:2454] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8adb8adeb1f43b992722da92bcce13d in term 0.
I20260810 23:01:20.979580 18746 leader_election.cc:302] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df4a4476bbd3417190567a6c72e00c5a, f8adb8adeb1f43b992722da92bcce13d; no voters:
I20260810 23:01:20.979899 19356 raft_consensus.cc:2781] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:20.979959 19356 raft_consensus.cc:482] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:20.979985 19356 raft_consensus.cc:3037] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:20.981009 19356 raft_consensus.cc:504] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:20.981359 19356 leader_election.cc:288] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [CANDIDATE]: Term 1 election: Requested vote from peers df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281), b7e8a19b8df54298bfb197f611be5134 (127.15.56.3:36885)
I20260810 23:01:20.981724 18625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c21c7e41d647f181a2a48637caae3f" candidate_uuid: "f8adb8adeb1f43b992722da92bcce13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4476bbd3417190567a6c72e00c5a"
I20260810 23:01:20.981724 19115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c21c7e41d647f181a2a48637caae3f" candidate_uuid: "f8adb8adeb1f43b992722da92bcce13d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7e8a19b8df54298bfb197f611be5134"
I20260810 23:01:20.981837 18625 raft_consensus.cc:3037] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:20.981840 19115 raft_consensus.cc:3037] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:20.982724 18625 raft_consensus.cc:2454] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8adb8adeb1f43b992722da92bcce13d in term 1.
I20260810 23:01:20.982878 19115 raft_consensus.cc:2454] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8adb8adeb1f43b992722da92bcce13d in term 1.
I20260810 23:01:20.982999 18746 leader_election.cc:302] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df4a4476bbd3417190567a6c72e00c5a, f8adb8adeb1f43b992722da92bcce13d; no voters:
I20260810 23:01:20.983166 19356 raft_consensus.cc:2781] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:20.983434 19356 raft_consensus.cc:686] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 1 LEADER]: Becoming Leader. State: Replica: f8adb8adeb1f43b992722da92bcce13d, State: Running, Role: LEADER
I20260810 23:01:20.983726 19356 consensus_queue.cc:237] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:20.986090 18368 catalog_manager.cc:5495] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d reported cstate change: term changed from 0 to 1, leader changed from <none> to f8adb8adeb1f43b992722da92bcce13d (127.15.56.2). New cstate: current_term: 1 leader_uuid: "f8adb8adeb1f43b992722da92bcce13d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } health_report { overall_health: UNKNOWN } } }
I20260810 23:01:21.090368 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:21.086578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.15.56.2:34973)
0810 23:01:21.086856 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:21.086863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:21.086874 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:01:21.086949 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:21.086951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:21.087041 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:21.087170 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:21.087428 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.088032 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.088890 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.089126 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.089223 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:21.089483 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:21.089960 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:21.090070 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:21.090136 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 23:01:21.090192 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
I20260810 23:01:21.101377 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:21.097985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:42631 (local address 127.15.56.3:36885)
0810 23:01:21.098145 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:21.098150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:21.098239 (+ 89us) server_negotiation.cc:407] Connection header received
0810 23:01:21.098421 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:21.098424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:21.098477 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:21.098539 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:21.098802 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.099329 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.100261 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.100498 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.100589 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:21.100908 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:21.100991 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:21.101050 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:21.101157 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 23:01:21.101218 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:01:21.101383 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:21.097806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:36885 (local address 127.15.56.2:42631)
0810 23:01:21.098144 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:21.098159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:21.098291 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:21.098561 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:21.098564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:21.098573 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:01:21.098694 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:21.098700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.099441 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.099445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:21.100137 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:21.100147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.100607 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.100610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:21.100706 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:21.100760 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:01:21.101069 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:01:21.101077 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:01:21.101136 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 23:01:21.101226 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":101,"threads_started":1}
I20260810 23:01:21.107839 18625 raft_consensus.cc:1261] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Refusing update from remote peer f8adb8adeb1f43b992722da92bcce13d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:21.107972 19115 raft_consensus.cc:1261] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Refusing update from remote peer f8adb8adeb1f43b992722da92bcce13d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:21.108325 19356 consensus_queue.cc:1035] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:21.108533 19367 consensus_queue.cc:1035] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:21.126730 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:21.123363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:55763 (local address 127.15.56.2:34973)
0810 23:01:21.123482 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:21.123487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:21.123680 (+ 193us) server_negotiation.cc:407] Connection header received
0810 23:01:21.123850 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:21.123854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:21.123902 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:21.123961 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:21.124201 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.124715 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.125677 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.126015 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.126125 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:21.126387 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:21.126455 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:21.126507 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:21.126574 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 23:01:21.126619 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:01:21.126762 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:21.123307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:34973 (local address 127.15.56.3:55763)
0810 23:01:21.123596 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:21.123608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:21.123752 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:21.123978 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:21.123981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:21.123990 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:01:21.124102 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:21.124108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.124818 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.124821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:21.125565 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:21.125575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.126139 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.126142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:21.126231 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:21.126261 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:01:21.126518 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:01:21.126523 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:01:21.126566 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 23:01:21.126636 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":118,"threads_started":1}
I20260810 23:01:21.154384 18368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:21.154542 18368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:21.156566 18368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850: Sending DeleteTablet for 3 replicas of tablet c9c21c7e41d647f181a2a48637caae3f
I20260810 23:01:21.157178 18836 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c21c7e41d647f181a2a48637caae3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:21 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:35882
I20260810 23:01:21.157378 19077 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c21c7e41d647f181a2a48637caae3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:21 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:47150
I20260810 23:01:21.157526 19409 tablet_replica.cc:330] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d: stopping tablet replica
I20260810 23:01:21.157500 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet c9c21c7e41d647f181a2a48637caae3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:21 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:51308
I20260810 23:01:21.157655 19409 raft_consensus.cc:2229] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:21.157958 19415 tablet_replica.cc:330] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a: stopping tablet replica
I20260810 23:01:21.158003 19409 raft_consensus.cc:2258] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:21.158145 19415 raft_consensus.cc:2229] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:21.158339 19415 raft_consensus.cc:2258] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:21.158627 18368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:01:21.158833 19409 ts_tablet_manager.cc:1905] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:21.159052 19415 ts_tablet_manager.cc:1905] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:01:21.159057 18368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:21.161860 19415 ts_tablet_manager.cc:1918] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:21.162179 19414 tablet_replica.cc:330] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134: stopping tablet replica
I20260810 23:01:21.162386 19414 raft_consensus.cc:2229] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:21.162580 19414 raft_consensus.cc:2258] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:21.162781 19415 log.cc:1196] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-0/wal/wals/c9c21c7e41d647f181a2a48637caae3f
I20260810 23:01:21.163296 19414 ts_tablet_manager.cc:1905] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:21.164063 18836 tablet_service.cc:1467] Processing CreateTablet for tablet 85ea025574ef40e5a5d92380a1429e37 (DEFAULT_TABLE table=test-table [id=e25fbb55494e40b696b50fc7f1e9fc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:21.164352 18836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ea025574ef40e5a5d92380a1429e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:21.164752 19077 tablet_service.cc:1467] Processing CreateTablet for tablet 85ea025574ef40e5a5d92380a1429e37 (DEFAULT_TABLE table=test-table [id=e25fbb55494e40b696b50fc7f1e9fc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:21.164971 19077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ea025574ef40e5a5d92380a1429e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:21.166661 19409 ts_tablet_manager.cc:1918] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:21.166759 19409 log.cc:1196] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/c9c21c7e41d647f181a2a48637caae3f
I20260810 23:01:21.167047 19409 ts_tablet_manager.cc:1939] T c9c21c7e41d647f181a2a48637caae3f P f8adb8adeb1f43b992722da92bcce13d: Deleting consensus metadata
I20260810 23:01:21.168141 18345 catalog_manager.cc:4841] TS f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973): tablet c9c21c7e41d647f181a2a48637caae3f (table test-table [id=02004f0fb89945568ff37676beb02f3b]) successfully deleted
I20260810 23:01:21.169672 19418 tablet_bootstrap.cc:492] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134: Bootstrap starting.
I20260810 23:01:21.164148 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 85ea025574ef40e5a5d92380a1429e37 (DEFAULT_TABLE table=test-table [id=e25fbb55494e40b696b50fc7f1e9fc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:21.169705 19414 ts_tablet_manager.cc:1918] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:21.170114 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ea025574ef40e5a5d92380a1429e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:21.170162 19414 log.cc:1196] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/c9c21c7e41d647f181a2a48637caae3f
I20260810 23:01:21.170421 19414 ts_tablet_manager.cc:1939] T c9c21c7e41d647f181a2a48637caae3f P b7e8a19b8df54298bfb197f611be5134: Deleting consensus metadata
I20260810 23:01:21.170674 19418 tablet_bootstrap.cc:654] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:21.171079 18347 catalog_manager.cc:4841] TS b7e8a19b8df54298bfb197f611be5134 (127.15.56.3:36885): tablet c9c21c7e41d647f181a2a48637caae3f (table test-table [id=02004f0fb89945568ff37676beb02f3b]) successfully deleted
I20260810 23:01:21.170116 19415 ts_tablet_manager.cc:1939] T c9c21c7e41d647f181a2a48637caae3f P df4a4476bbd3417190567a6c72e00c5a: Deleting consensus metadata
I20260810 23:01:21.171764 19418 tablet_bootstrap.cc:492] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134: No bootstrap required, opened a new log
I20260810 23:01:21.171842 19418 ts_tablet_manager.cc:1397] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:21.172071 19417 tablet_bootstrap.cc:492] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d: Bootstrap starting.
I20260810 23:01:21.172231 19418 raft_consensus.cc:348] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:21.172349 19418 raft_consensus.cc:374] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:21.172380 19418 raft_consensus.cc:729] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e8a19b8df54298bfb197f611be5134, State: Initialized, Role: FOLLOWER
I20260810 23:01:21.172505 19418 consensus_queue.cc:260] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:21.172780 19418 ts_tablet_manager.cc:1428] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:21.172878 18343 catalog_manager.cc:4841] TS df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281): tablet c9c21c7e41d647f181a2a48637caae3f (table test-table [id=02004f0fb89945568ff37676beb02f3b]) successfully deleted
I20260810 23:01:21.173054 19417 tablet_bootstrap.cc:654] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:21.173897 19423 tablet_bootstrap.cc:492] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a: Bootstrap starting.
I20260810 23:01:21.174808 19423 tablet_bootstrap.cc:654] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:21.175967 19423 tablet_bootstrap.cc:492] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a: No bootstrap required, opened a new log
I20260810 23:01:21.176050 19423 ts_tablet_manager.cc:1397] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:01:21.176491 19423 raft_consensus.cc:348] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:21.176610 19423 raft_consensus.cc:374] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:21.176667 19423 raft_consensus.cc:729] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4a4476bbd3417190567a6c72e00c5a, State: Initialized, Role: FOLLOWER
I20260810 23:01:21.176808 19423 consensus_queue.cc:260] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:21.177157 19423 ts_tablet_manager.cc:1428] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:01:21.177726 19417 tablet_bootstrap.cc:492] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d: No bootstrap required, opened a new log
I20260810 23:01:21.177871 19417 ts_tablet_manager.cc:1397] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 23:01:21.178396 19417 raft_consensus.cc:348] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:21.178519 19417 raft_consensus.cc:374] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:21.178550 19417 raft_consensus.cc:729] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8adb8adeb1f43b992722da92bcce13d, State: Initialized, Role: FOLLOWER
I20260810 23:01:21.178696 19417 consensus_queue.cc:260] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:21.178959 19417 ts_tablet_manager.cc:1428] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:21.189172 19328 raft_consensus.cc:482] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:21.189285 19328 raft_consensus.cc:504] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:21.189680 19328 leader_election.cc:288] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281), f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973)
I20260810 23:01:21.190032 18625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ea025574ef40e5a5d92380a1429e37" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4476bbd3417190567a6c72e00c5a" is_pre_election: true
I20260810 23:01:21.190057 18868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ea025574ef40e5a5d92380a1429e37" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8adb8adeb1f43b992722da92bcce13d" is_pre_election: true
I20260810 23:01:21.190183 18625 raft_consensus.cc:2454] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 0.
I20260810 23:01:21.190181 18868 raft_consensus.cc:2454] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 0.
I20260810 23:01:21.190474 18997 leader_election.cc:302] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7e8a19b8df54298bfb197f611be5134, df4a4476bbd3417190567a6c72e00c5a; no voters:
I20260810 23:01:21.190699 19328 raft_consensus.cc:2781] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:21.190774 19328 raft_consensus.cc:482] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:21.190821 19328 raft_consensus.cc:3037] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:21.192022 19328 raft_consensus.cc:504] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:21.192314 19328 leader_election.cc:288] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 election: Requested vote from peers df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281), f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973)
I20260810 23:01:21.192607 18625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ea025574ef40e5a5d92380a1429e37" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4a4476bbd3417190567a6c72e00c5a"
I20260810 23:01:21.192644 18868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ea025574ef40e5a5d92380a1429e37" candidate_uuid: "b7e8a19b8df54298bfb197f611be5134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8adb8adeb1f43b992722da92bcce13d"
I20260810 23:01:21.192727 18625 raft_consensus.cc:3037] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:21.192768 18868 raft_consensus.cc:3037] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:21.193593 18868 raft_consensus.cc:2454] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 1.
I20260810 23:01:21.193814 18625 raft_consensus.cc:2454] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7e8a19b8df54298bfb197f611be5134 in term 1.
I20260810 23:01:21.193892 18998 leader_election.cc:302] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7e8a19b8df54298bfb197f611be5134, f8adb8adeb1f43b992722da92bcce13d; no voters:
I20260810 23:01:21.194137 19328 raft_consensus.cc:2781] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:21.194228 19328 raft_consensus.cc:686] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 1 LEADER]: Becoming Leader. State: Replica: b7e8a19b8df54298bfb197f611be5134, State: Running, Role: LEADER
I20260810 23:01:21.194361 19328 consensus_queue.cc:237] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } }
I20260810 23:01:21.195981 18370 catalog_manager.cc:5495] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7e8a19b8df54298bfb197f611be5134 (127.15.56.3). New cstate: current_term: 1 leader_uuid: "b7e8a19b8df54298bfb197f611be5134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7e8a19b8df54298bfb197f611be5134" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260810 23:01:21.210245 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:21.206598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.15.56.3:36885)
0810 23:01:21.206703 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:21.206708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:21.206749 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:01:21.206879 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:21.206882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:21.206930 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:21.206998 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:21.207362 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.207994 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.209232 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:21.209487 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:21.209566 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:21.209815 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:21.209914 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:21.209974 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:21.210069 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 23:01:21.210113 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:01:21.216198 18625 raft_consensus.cc:1261] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a [term 1 FOLLOWER]: Refusing update from remote peer b7e8a19b8df54298bfb197f611be5134: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:21.216552 18868 raft_consensus.cc:1261] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Refusing update from remote peer b7e8a19b8df54298bfb197f611be5134: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:21.216917 19328 consensus_queue.cc:1035] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8adb8adeb1f43b992722da92bcce13d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:21.217522 19328 consensus_queue.cc:1035] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4a4476bbd3417190567a6c72e00c5a" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:01:21.232137 19372 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:01:22.783387 18370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:22.783551 18370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:22.786109 18370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 675a6643931244b3af073a82cbb8d850: Sending DeleteTablet for 3 replicas of tablet 85ea025574ef40e5a5d92380a1429e37
I20260810 23:01:22.786763 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ea025574ef40e5a5d92380a1429e37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:22 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:51308
I20260810 23:01:22.786823 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ea025574ef40e5a5d92380a1429e37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:22 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:47150
I20260810 23:01:22.786819 18836 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ea025574ef40e5a5d92380a1429e37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:22 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:35882
I20260810 23:01:22.787256 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 18479
I20260810 23:01:22.787273 19464 tablet_replica.cc:330] T 85ea025574ef40e5a5d92380a1429e37 P df4a4476bbd3417190567a6c72e00c5a: stopping tablet replica
I20260810 23:01:22.788007 19465 tablet_replica.cc:330] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134: stopping tablet replica
I20260810 23:01:22.788136 19465 raft_consensus.cc:2229] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:22.788363 19465 raft_consensus.cc:2258] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:22.788618 19467 tablet_replica.cc:330] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d: stopping tablet replica
I20260810 23:01:22.788746 19467 raft_consensus.cc:2229] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:22.788861 19467 raft_consensus.cc:2258] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:22.791638 19467 ts_tablet_manager.cc:1905] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:22.793956 19467 ts_tablet_manager.cc:1918] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:01:22.794040 19467 log.cc:1196] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/85ea025574ef40e5a5d92380a1429e37
I20260810 23:01:22.794268 19467 ts_tablet_manager.cc:1939] T 85ea025574ef40e5a5d92380a1429e37 P f8adb8adeb1f43b992722da92bcce13d: Deleting consensus metadata
I20260810 23:01:22.794871 19465 ts_tablet_manager.cc:1905] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:22.795212 18345 catalog_manager.cc:4841] TS f8adb8adeb1f43b992722da92bcce13d (127.15.56.2:34973): tablet 85ea025574ef40e5a5d92380a1429e37 (table test-table [id=e25fbb55494e40b696b50fc7f1e9fc15]) successfully deleted
I20260810 23:01:22.797207 19465 ts_tablet_manager.cc:1918] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:01:22.797297 19465 log.cc:1196] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/85ea025574ef40e5a5d92380a1429e37
I20260810 23:01:22.797540 19465 ts_tablet_manager.cc:1939] T 85ea025574ef40e5a5d92380a1429e37 P b7e8a19b8df54298bfb197f611be5134: Deleting consensus metadata
W20260810 23:01:22.798141 18343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:45281 (error 108)
I20260810 23:01:22.798538 18347 catalog_manager.cc:4841] TS b7e8a19b8df54298bfb197f611be5134 (127.15.56.3:36885): tablet 85ea025574ef40e5a5d92380a1429e37 (table test-table [id=e25fbb55494e40b696b50fc7f1e9fc15]) successfully deleted
I20260810 23:01:22.799108 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 18720
I20260810 23:01:22.799412 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:22.798917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:45281 (local address 127.0.0.1:51324)
0810 23:01:22.799219 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:22.799305 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":107,"threads_started":1}
W20260810 23:01:22.810873 18343 catalog_manager.cc:4584] TS df4a4476bbd3417190567a6c72e00c5a (127.15.56.1:45281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85ea025574ef40e5a5d92380a1429e37: Network error: Client connection negotiation failed: client connection to 127.15.56.1:45281: connect: Connection refused (error 111)
I20260810 23:01:22.819154 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 18975
I20260810 23:01:22.832494 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 18320
2026-08-10T23:01:22Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6921 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:22 dist-test-slave-6bbx krb5kdc[19476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:22 dist-test-slave-6bbx krb5kdc[19476](info): set up 2 sockets
Aug 10 23:01:22 dist-test-slave-6bbx krb5kdc[19476](info): commencing operation
krb5kdc: starting...
W20260810 23:01:23.996446 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:24 dist-test-slave-6bbx krb5kdc[19476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:24.120510 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:42467
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:42467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:24.236090 19504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:24.236384 19504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:24.236442 19504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:24.240448 19504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:24.240511 19504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:24.240537 19504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:24.240559 19504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:24.240587 19504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:24.245986 19504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:42467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:42467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:24.247108 19504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:24.248250 19504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:24.257339 19515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:24.257395 19504 server_base.cc:1034] running on GCE node
W20260810 23:01:24.257334 19511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:24.257375 19512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:24.258165 19504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:24.259339 19504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:24.260501 19504 hybrid_clock.cc:648] HybridClock initialized: now 1786402884260485 us; error 27 us; skew 500 ppm
Aug 10 23:01:24 dist-test-slave-6bbx krb5kdc[19476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402884, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:24.267078 19504 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:24.268076 19504 webserver.cc:460] Webserver started at http://127.15.56.62:41105/ using document root <none> and password file <none>
I20260810 23:01:24.268641 19504 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:24.268710 19504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:24.268932 19504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:24.271044 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "45e040d1d4184db281654f9219cbe807"
format_stamp: "Formatted at 2026-08-10 23:01:24 on dist-test-slave-6bbx"
server_key: "c7f596b9112e2d8824a6b757f68e3a17"
server_key_iv: "d3a858be1b1ea6236d087507cb277eb1"
server_key_version: "encryptionkey@0"
I20260810 23:01:24.271612 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "45e040d1d4184db281654f9219cbe807"
format_stamp: "Formatted at 2026-08-10 23:01:24 on dist-test-slave-6bbx"
server_key: "c7f596b9112e2d8824a6b757f68e3a17"
server_key_iv: "d3a858be1b1ea6236d087507cb277eb1"
server_key_version: "encryptionkey@0"
I20260810 23:01:24.275323 19504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:24.277896 19529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:24.279206 19504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:24.279323 19504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "45e040d1d4184db281654f9219cbe807"
format_stamp: "Formatted at 2026-08-10 23:01:24 on dist-test-slave-6bbx"
server_key: "c7f596b9112e2d8824a6b757f68e3a17"
server_key_iv: "d3a858be1b1ea6236d087507cb277eb1"
server_key_version: "encryptionkey@0"
I20260810 23:01:24.279423 19504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:24.324177 19504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:24.327545 19504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:24.327761 19504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:24.334906 19504 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:42467
I20260810 23:01:24.334944 19632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:42467 every 8 connection(s)
I20260810 23:01:24.335971 19504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:24.337641 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 19504
I20260810 23:01:24.337733 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:24.337994 15584 external_mini_cluster.cc:1428] Setting key eddfbc933b0407a20e8c9d7ddca4103d
I20260810 23:01:24.339130 19633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:24.346354 19633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807: Bootstrap starting.
Aug 10 23:01:24 dist-test-slave-6bbx krb5kdc[19476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:24.348918 19633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:24.349838 19633 log.cc:824] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:24.351895 19633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807: No bootstrap required, opened a new log
I20260810 23:01:24.353055 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:24.339547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.15.56.62:42467)
0810 23:01:24.340022 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:24.340048 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:24.340092 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:01:24.340947 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:24.340972 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:24.341443 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:24.341826 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:24.342557 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:24.344069 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:24.344968 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:24.345687 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:24.345847 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:24.348337 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:24.348343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:24.348361 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:24.348392 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:24.350339 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:24.350765 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:24.350771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:24.350777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:24.350891 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:24.351153 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:24.351156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:24.351158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:24.351554 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:24.351763 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:24.352041 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 23:01:24.352338 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":164,"threads_started":1}
I20260810 23:01:24.355854 19633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e040d1d4184db281654f9219cbe807" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42467 } }
I20260810 23:01:24.356089 19633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:24.356150 19633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e040d1d4184db281654f9219cbe807, State: Initialized, Role: FOLLOWER
I20260810 23:01:24.356575 19633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e040d1d4184db281654f9219cbe807" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42467 } }
I20260810 23:01:24.356760 19633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:24.356832 19633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:24.356937 19633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:24.357949 19633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e040d1d4184db281654f9219cbe807" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42467 } }
I20260810 23:01:24.358250 19633 leader_election.cc:302] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45e040d1d4184db281654f9219cbe807; no voters:
I20260810 23:01:24.358549 19633 leader_election.cc:288] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:24.358685 19641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:24.358920 19641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [term 1 LEADER]: Becoming Leader. State: Replica: 45e040d1d4184db281654f9219cbe807, State: Running, Role: LEADER
I20260810 23:01:24.359213 19641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e040d1d4184db281654f9219cbe807" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42467 } }
I20260810 23:01:24.359760 19633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:24.360824 19642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45e040d1d4184db281654f9219cbe807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e040d1d4184db281654f9219cbe807" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42467 } } }
I20260810 23:01:24.360958 19642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:24.361354 19656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:24.361320 19643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45e040d1d4184db281654f9219cbe807. Latest consensus state: current_term: 1 leader_uuid: "45e040d1d4184db281654f9219cbe807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45e040d1d4184db281654f9219cbe807" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42467 } } }
I20260810 23:01:24.361467 19643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:24.364486 19656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:24.370174 19656 catalog_manager.cc:1347] Generated new cluster ID: 599918d9d9ca4d6c9d9873b8aec84d10
I20260810 23:01:24.370252 19656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:24.379067 19656 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:24.380362 19656 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:24.391213 19656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45e040d1d4184db281654f9219cbe807: Generated new TSK 0
I20260810 23:01:24.392220 19656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:01:24.426677 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42467
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:24.542177 19680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:24.542451 19680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:24.542493 19680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:24.546540 19680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:24.546605 19680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:24.546684 19680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:24.551964 19680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:24.553061 19680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:24.554113 19680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:24.563903 19690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:24.563896 19686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:24.563984 19680 server_base.cc:1034] running on GCE node
W20260810 23:01:24.564113 19688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:24.564507 19680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:24.565148 19680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:24.566315 19680 hybrid_clock.cc:648] HybridClock initialized: now 1786402884566294 us; error 33 us; skew 500 ppm
Aug 10 23:01:24 dist-test-slave-6bbx krb5kdc[19476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402884, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:24.569098 19680 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:01:24.569958 19680 webserver.cc:460] Webserver started at http://127.15.56.1:42633/ using document root <none> and password file <none>
I20260810 23:01:24.570516 19680 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:24.570580 19680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:24.570783 19680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:24.572868 19680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "eac9eb32f2694e768637fcb4406625c7"
format_stamp: "Formatted at 2026-08-10 23:01:24 on dist-test-slave-6bbx"
server_key: "83e287a78ad09601afa8050ce980c56b"
server_key_iv: "4c53cffdc33961e58d9e9bbd3bb1d793"
server_key_version: "encryptionkey@0"
I20260810 23:01:24.573427 19680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "eac9eb32f2694e768637fcb4406625c7"
format_stamp: "Formatted at 2026-08-10 23:01:24 on dist-test-slave-6bbx"
server_key: "83e287a78ad09601afa8050ce980c56b"
server_key_iv: "4c53cffdc33961e58d9e9bbd3bb1d793"
server_key_version: "encryptionkey@0"
I20260810 23:01:24.577030 19680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:01:24.579545 19704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:24.580864 19680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:24.580978 19680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "eac9eb32f2694e768637fcb4406625c7"
format_stamp: "Formatted at 2026-08-10 23:01:24 on dist-test-slave-6bbx"
server_key: "83e287a78ad09601afa8050ce980c56b"
server_key_iv: "4c53cffdc33961e58d9e9bbd3bb1d793"
server_key_version: "encryptionkey@0"
I20260810 23:01:24.581082 19680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:24.590992 19680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:24.593935 19680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:24.594105 19680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:24.594702 19680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:24.595546 19680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:24.595593 19680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:24.595634 19680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:24.595717 19680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:24.604562 19680 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:38281
I20260810 23:01:24.604592 19909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:38281 every 8 connection(s)
I20260810 23:01:24.605499 19680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:24.613893 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 19680
I20260810 23:01:24.613999 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:24.614295 15584 external_mini_cluster.cc:1428] Setting key a9c8ad8da0fabc2b85822f26c3aaef41
Aug 10 23:01:24 dist-test-slave-6bbx krb5kdc[19476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402884, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:24.618889 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:24.607360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:60275 (local address 127.15.56.62:42467)
0810 23:01:24.607493 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:24.607496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:24.608570 (+ 1074us) server_negotiation.cc:407] Connection header received
0810 23:01:24.609466 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:24.609470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:24.609517 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:24.609591 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:24.610774 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:24.611321 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:24.612591 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:24.612828 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:24.612911 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:24.615387 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:24.615391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:24.615394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:24.615419 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:24.616800 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:24.617373 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:24.617375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:24.617376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:24.617465 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:24.617813 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:24.617816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:24.617818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:24.617997 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:24.618081 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:24.618590 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 23:01:24.618727 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:01:24.619786 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:24.607727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:42467 (local address 127.15.56.1:60275)
0810 23:01:24.608423 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:24.608467 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:24.609258 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:24.609759 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:24.609766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:24.610127 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:24.610613 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:24.610627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:24.611461 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:24.611465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:24.612466 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:24.612477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:24.612962 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:24.612964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:24.613116 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:24.613725 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:24.613742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:24.615231 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:24.616916 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:24.616923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:24.616940 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:24.617286 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:24.617560 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:24.617563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:24.617565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:24.617719 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:24.618098 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:24.618105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:24.618357 (+ 252us) client_negotiation.cc:769] Sending connection context
0810 23:01:24.618533 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 23:01:24.618884 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":117,"threads_started":1}
I20260810 23:01:24.621001 19910 heartbeater.cc:347] Connected to a master server at 127.15.56.62:42467
I20260810 23:01:24.621243 19910 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:24.621734 19910 heartbeater.cc:511] Master 127.15.56.62:42467 requested a full tablet report, sending...
I20260810 23:01:24.623420 19563 ts_manager.cc:194] Registered new tserver with Master: eac9eb32f2694e768637fcb4406625c7 (127.15.56.1:38281)
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:01:24.625073 19563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:60275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:01:24.642206 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42467
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:24.759990 19921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:24.760334 19921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:24.760401 19921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:24.764779 19921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:24.764859 19921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:24.764925 19921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:24.770535 19921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:24.771804 19921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:24.772954 19921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:24.783355 19932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:24.783341 19931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:24.783341 19935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:24.783739 19921 server_base.cc:1034] running on GCE node
I20260810 23:01:24.784262 19921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:24.785010 19921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:24.786280 19921 hybrid_clock.cc:648] HybridClock initialized: now 1786402884786256 us; error 40 us; skew 500 ppm
Aug 10 23:01:24 dist-test-slave-6bbx krb5kdc[19476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402884, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:24.789408 19921 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:01:24.790463 19921 webserver.cc:460] Webserver started at http://127.15.56.2:33039/ using document root <none> and password file <none>
I20260810 23:01:24.791047 19921 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:24.791109 19921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:24.791308 19921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:24.793561 19921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "d604ce07ccae4d46851dd87626455b5d"
format_stamp: "Formatted at 2026-08-10 23:01:24 on dist-test-slave-6bbx"
server_key: "257c73af7ae57e44485c6ffa19d14d66"
server_key_iv: "57e38b8608ca800c25f962ea10580283"
server_key_version: "encryptionkey@0"
I20260810 23:01:24.794310 19921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "d604ce07ccae4d46851dd87626455b5d"
format_stamp: "Formatted at 2026-08-10 23:01:24 on dist-test-slave-6bbx"
server_key: "257c73af7ae57e44485c6ffa19d14d66"
server_key_iv: "57e38b8608ca800c25f962ea10580283"
server_key_version: "encryptionkey@0"
I20260810 23:01:24.798962 19921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 23:01:24.801812 19948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:24.803201 19921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:01:24.803337 19921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "d604ce07ccae4d46851dd87626455b5d"
format_stamp: "Formatted at 2026-08-10 23:01:24 on dist-test-slave-6bbx"
server_key: "257c73af7ae57e44485c6ffa19d14d66"
server_key_iv: "57e38b8608ca800c25f962ea10580283"
server_key_version: "encryptionkey@0"
I20260810 23:01:24.803457 19921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:24.828140 19921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:24.831442 19921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:24.831633 19921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:24.832324 19921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:24.833191 19921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:24.833241 19921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:24.833292 19921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:24.833320 19921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:24.842214 19921 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:34271
I20260810 23:01:24.842304 20135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:34271 every 8 connection(s)
I20260810 23:01:24.843149 19921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:24.849747 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 19921
I20260810 23:01:24.849872 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:24.850167 15584 external_mini_cluster.cc:1428] Setting key 0f56598550cf546e627645d033fb674c
Aug 10 23:01:24 dist-test-slave-6bbx krb5kdc[19476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402884, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:24.857925 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:24.845048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:47155 (local address 127.15.56.62:42467)
0810 23:01:24.845204 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:24.845207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:24.846274 (+ 1067us) server_negotiation.cc:407] Connection header received
0810 23:01:24.847173 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:24.847177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:24.847238 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:24.847314 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:24.848578 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:24.849112 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:24.850636 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:24.850930 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:24.851034 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:24.853931 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:24.853935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:24.853939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:24.853970 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:24.855563 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:24.856111 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:24.856115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:24.856117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:24.856213 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:24.856595 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:24.856599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:24.856601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:24.856857 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:24.856965 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:24.857579 (+ 614us) server_negotiation.cc:299] Negotiation successful
0810 23:01:24.857747 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:01:24.858647 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:24.845408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:42467 (local address 127.15.56.2:47155)
0810 23:01:24.846094 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:24.846137 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:24.846962 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:24.847502 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:24.847510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:24.847928 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:24.848433 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:24.848448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:24.849234 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:24.849237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:24.850480 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:24.850492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:24.851083 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:24.851086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:24.851264 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:24.851914 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:24.851935 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:24.853746 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:24.855721 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:24.855728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:24.855744 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:24.856009 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:24.856315 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:24.856318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:24.856321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:24.856484 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:24.857051 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:24.857056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:24.857312 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 23:01:24.857473 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:01:24.857757 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":107,"threads_started":1}
I20260810 23:01:24.859898 20136 heartbeater.cc:347] Connected to a master server at 127.15.56.62:42467
I20260810 23:01:24.860140 20136 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:24.860728 20136 heartbeater.cc:511] Master 127.15.56.62:42467 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:01:24.861810 19563 ts_manager.cc:194] Registered new tserver with Master: d604ce07ccae4d46851dd87626455b5d (127.15.56.2:34271)
I20260810 23:01:24.862511 19563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:47155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:01:24.878568 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42467
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:24.995183 20142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:24.995465 20142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:24.995510 20142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:24.999507 20142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:24.999573 20142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:24.999629 20142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:25.004962 20142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:25.005973 20142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:25.006996 20142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:25.016937 20154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:25.016988 20142 server_base.cc:1034] running on GCE node
W20260810 23:01:25.016937 20153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:25.016938 20159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:25.017534 20142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:25.018186 20142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:25.019351 20142 hybrid_clock.cc:648] HybridClock initialized: now 1786402885019321 us; error 40 us; skew 500 ppm
Aug 10 23:01:25 dist-test-slave-6bbx krb5kdc[19476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402885, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:25.022121 20142 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:01:25.023052 20142 webserver.cc:460] Webserver started at http://127.15.56.3:40351/ using document root <none> and password file <none>
I20260810 23:01:25.023597 20142 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:25.023684 20142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:25.023913 20142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:25.025959 20142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "c0567e17832548c08acc3dff141350cb"
format_stamp: "Formatted at 2026-08-10 23:01:25 on dist-test-slave-6bbx"
server_key: "60c68ed861f687fea2af283e4887ec2a"
server_key_iv: "1ef568f6302efba56827851783c95a85"
server_key_version: "encryptionkey@0"
I20260810 23:01:25.026495 20142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "c0567e17832548c08acc3dff141350cb"
format_stamp: "Formatted at 2026-08-10 23:01:25 on dist-test-slave-6bbx"
server_key: "60c68ed861f687fea2af283e4887ec2a"
server_key_iv: "1ef568f6302efba56827851783c95a85"
server_key_version: "encryptionkey@0"
I20260810 23:01:25.030445 20142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 23:01:25.033270 20173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:25.034544 20142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:01:25.034672 20142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "c0567e17832548c08acc3dff141350cb"
format_stamp: "Formatted at 2026-08-10 23:01:25 on dist-test-slave-6bbx"
server_key: "60c68ed861f687fea2af283e4887ec2a"
server_key_iv: "1ef568f6302efba56827851783c95a85"
server_key_version: "encryptionkey@0"
I20260810 23:01:25.034780 20142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:25.047122 20142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:25.050172 20142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:25.050371 20142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:25.050981 20142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:25.051879 20142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:25.051931 20142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:25.051987 20142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:25.052011 20142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:25.060570 20142 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:42701
I20260810 23:01:25.060603 20372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:42701 every 8 connection(s)
I20260810 23:01:25.061537 20142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:25.066013 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 20142
I20260810 23:01:25.066128 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:25.066401 15584 external_mini_cluster.cc:1428] Setting key 4aeca4f24bdcadd48885021462adc600
Aug 10 23:01:25 dist-test-slave-6bbx krb5kdc[19476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402885, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:25.075090 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:25.063435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:38283 (local address 127.15.56.62:42467)
0810 23:01:25.063580 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:25.063584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:25.064765 (+ 1181us) server_negotiation.cc:407] Connection header received
0810 23:01:25.065676 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:25.065679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:25.065737 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:25.065822 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:25.067163 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:25.067889 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:25.069089 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:25.069308 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:25.069397 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:25.071720 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:25.071723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:25.071725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:25.071750 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:25.073131 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:25.073592 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:25.073595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:25.073596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:25.073671 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:25.073947 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:25.073949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:25.073950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:25.074122 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:25.074195 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:25.074801 (+ 606us) server_negotiation.cc:299] Negotiation successful
0810 23:01:25.074933 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:01:25.076274 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:25.063895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:42467 (local address 127.15.56.3:38283)
0810 23:01:25.064614 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:25.064660 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:25.065467 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:25.066005 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:25.066011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:25.066418 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:25.067031 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:25.067048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:25.068011 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:25.068014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:25.068976 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:25.068987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:25.069440 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:25.069443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:25.069588 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:25.070130 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:25.070147 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:25.071547 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:25.073238 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:25.073243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:25.073258 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:25.073510 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:25.073737 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:25.073739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:25.073740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:25.073880 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:25.074194 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:25.074199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:25.074534 (+ 335us) client_negotiation.cc:769] Sending connection context
0810 23:01:25.074779 (+ 245us) client_negotiation.cc:240] Negotiation successful
0810 23:01:25.075144 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":112,"threads_started":1}
I20260810 23:01:25.079257 20374 heartbeater.cc:347] Connected to a master server at 127.15.56.62:42467
I20260810 23:01:25.079540 20374 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:25.080183 20374 heartbeater.cc:511] Master 127.15.56.62:42467 requested a full tablet report, sending...
I20260810 23:01:25.081465 19563 ts_manager.cc:194] Registered new tserver with Master: c0567e17832548c08acc3dff141350cb (127.15.56.3:42701)
I20260810 23:01:25.082194 19563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:38283
I20260810 23:01:25.083511 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:01:25.097522 19639 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:25.095997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.15.56.62:42467)
0810 23:01:25.096195 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:25.096200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:25.096350 (+ 150us) server_negotiation.cc:407] Connection header received
0810 23:01:25.096492 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:25.096496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:25.096544 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:25.096619 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:25.097400 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36128: BlockingRecv error: recv got EOF from 127.0.0.1:36128 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:01:25.098587 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 19680
I20260810 23:01:25.115060 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 19921
I20260810 23:01:25.128123 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 20142
I20260810 23:01:25.153343 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 19504
2026-08-10T23:01:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2318 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:25 dist-test-slave-6bbx krb5kdc[20389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:25 dist-test-slave-6bbx krb5kdc[20389](info): set up 2 sockets
Aug 10 23:01:25 dist-test-slave-6bbx krb5kdc[20389](info): commencing operation
krb5kdc: starting...
W20260810 23:01:26.336823 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:26 dist-test-slave-6bbx krb5kdc[20389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:26.466661 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33925
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:35801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:33925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:26.580080 20410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:26.580353 20410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:26.580408 20410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:26.584457 20410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:26.584522 20410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:26.584547 20410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:26.584569 20410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:26.584599 20410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:26.589772 20410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:35801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:33925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:26.590821 20410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:26.591914 20410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:26.600070 20416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:26.600081 20420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:26.600126 20417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:26.600162 20410 server_base.cc:1034] running on GCE node
I20260810 23:01:26.600919 20410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:26.602032 20410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:26.603185 20410 hybrid_clock.cc:648] HybridClock initialized: now 1786402886603165 us; error 31 us; skew 500 ppm
Aug 10 23:01:26 dist-test-slave-6bbx krb5kdc[20389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402886, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:26.606004 20410 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:26.606940 20410 webserver.cc:460] Webserver started at http://127.15.56.62:35187/ using document root <none> and password file <none>
I20260810 23:01:26.607496 20410 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:26.607563 20410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:26.607827 20410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:26.609915 20410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "c8e05f94d97b40ebbda8a5bd8e4444a4"
format_stamp: "Formatted at 2026-08-10 23:01:26 on dist-test-slave-6bbx"
server_key: "6656e863b72caa95fe7da1625316df4c"
server_key_iv: "e9c0b3af0f8bed01779312949ad366cc"
server_key_version: "encryptionkey@0"
I20260810 23:01:26.610489 20410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "c8e05f94d97b40ebbda8a5bd8e4444a4"
format_stamp: "Formatted at 2026-08-10 23:01:26 on dist-test-slave-6bbx"
server_key: "6656e863b72caa95fe7da1625316df4c"
server_key_iv: "e9c0b3af0f8bed01779312949ad366cc"
server_key_version: "encryptionkey@0"
I20260810 23:01:26.614306 20410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:01:26.616729 20435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:26.618006 20410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:01:26.618156 20410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "c8e05f94d97b40ebbda8a5bd8e4444a4"
format_stamp: "Formatted at 2026-08-10 23:01:26 on dist-test-slave-6bbx"
server_key: "6656e863b72caa95fe7da1625316df4c"
server_key_iv: "e9c0b3af0f8bed01779312949ad366cc"
server_key_version: "encryptionkey@0"
I20260810 23:01:26.618265 20410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:26.641106 20410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:26.644495 20410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:26.644686 20410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:26.651369 20410 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:33925
I20260810 23:01:26.651613 20535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:33925 every 8 connection(s)
I20260810 23:01:26.652381 20410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:26.654026 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 20410
I20260810 23:01:26.654145 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:26.654398 15584 external_mini_cluster.cc:1428] Setting key 4c7cc2499d0680bfd4578b48793cf566
I20260810 23:01:26.655411 20538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:26.662190 20538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4: Bootstrap starting.
Aug 10 23:01:26 dist-test-slave-6bbx krb5kdc[20389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:26.664688 20538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:26.665544 20538 log.cc:824] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:26.667716 20538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4: No bootstrap required, opened a new log
I20260810 23:01:26.669130 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:26.655724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56176 (local address 127.15.56.62:33925)
0810 23:01:26.656164 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:26.656194 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:26.656237 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:01:26.657052 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:26.657075 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:26.657499 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:26.657867 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:26.658566 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:26.660028 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:26.660912 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:26.661601 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:26.661762 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:26.664329 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:26.664335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:26.664350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:26.664380 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:26.666218 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:26.666611 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:26.666616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:26.666622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:26.666759 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:26.666979 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:26.666981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:26.666983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:26.667475 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:26.667693 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:26.667891 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 23:01:26.668203 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":131,"threads_started":1}
I20260810 23:01:26.672070 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:01:26.671962 20538 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e05f94d97b40ebbda8a5bd8e4444a4" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33925 } }
I20260810 23:01:26.672219 20538 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:26.672276 20538 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e05f94d97b40ebbda8a5bd8e4444a4, State: Initialized, Role: FOLLOWER
I20260810 23:01:26.672700 20538 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e05f94d97b40ebbda8a5bd8e4444a4" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33925 } }
I20260810 23:01:26.672824 20538 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:26.672883 20538 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:26.672977 20538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:26.673841 20538 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e05f94d97b40ebbda8a5bd8e4444a4" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33925 } }
I20260810 23:01:26.674180 20538 leader_election.cc:302] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8e05f94d97b40ebbda8a5bd8e4444a4; no voters:
I20260810 23:01:26.674469 20538 leader_election.cc:288] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:26.674571 20553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:26.674753 20553 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [term 1 LEADER]: Becoming Leader. State: Replica: c8e05f94d97b40ebbda8a5bd8e4444a4, State: Running, Role: LEADER
I20260810 23:01:26.675048 20553 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e05f94d97b40ebbda8a5bd8e4444a4" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33925 } }
I20260810 23:01:26.675827 20538 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:26.676321 20553 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e05f94d97b40ebbda8a5bd8e4444a4. Latest consensus state: current_term: 1 leader_uuid: "c8e05f94d97b40ebbda8a5bd8e4444a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e05f94d97b40ebbda8a5bd8e4444a4" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33925 } } }
I20260810 23:01:26.676436 20553 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:26.676826 20559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:26.676865 20556 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8e05f94d97b40ebbda8a5bd8e4444a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e05f94d97b40ebbda8a5bd8e4444a4" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33925 } } }
I20260810 23:01:26.677178 20556 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:26.680326 20559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:26.686663 20559 catalog_manager.cc:1347] Generated new cluster ID: dae1d2ed0eea4deca4cc83a8eef77861
I20260810 23:01:26.686741 20559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:26.695083 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:26.689946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.15.56.62:33925)
0810 23:01:26.690055 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:26.690060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:26.690076 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:01:26.690119 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:26.690122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:26.690164 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:26.690242 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:26.690831 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:26.691589 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:26.692781 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:26.693109 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:26.693191 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:26.694093 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:26.694097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:26.694100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:26.694654 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:26.694688 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:26.694707 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:26.694794 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:26.694879 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 23:01:26.694940 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:01:26.705801 20559 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:26.706857 20559 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:26.716576 20559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8e05f94d97b40ebbda8a5bd8e4444a4: Generated new TSK 0
I20260810 23:01:26.717468 20559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:01:26.729167 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 20410
2026-08-10T23:01:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1574 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:26 dist-test-slave-6bbx krb5kdc[20596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:26 dist-test-slave-6bbx krb5kdc[20596](info): set up 2 sockets
Aug 10 23:01:26 dist-test-slave-6bbx krb5kdc[20596](info): commencing operation
krb5kdc: starting...
W20260810 23:01:27.889509 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:27 dist-test-slave-6bbx krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:28.022408 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:42137
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:42137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:28.141650 20617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:28.141973 20617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:28.142014 20617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:28.146334 20617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:28.146435 20617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:28.146454 20617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:28.146471 20617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:28.146485 20617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:28.152467 20617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:42137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:42137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:28.153765 20617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:28.154910 20617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:28.163849 20634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:28.163939 20617 server_base.cc:1034] running on GCE node
W20260810 23:01:28.163897 20632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:28.163898 20638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:28.164670 20617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:28.165848 20617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:28.167035 20617 hybrid_clock.cc:648] HybridClock initialized: now 1786402888167014 us; error 30 us; skew 500 ppm
Aug 10 23:01:28 dist-test-slave-6bbx krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:28.170220 20617 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:28.171271 20617 webserver.cc:460] Webserver started at http://127.15.56.62:38263/ using document root <none> and password file <none>
I20260810 23:01:28.171886 20617 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:28.171959 20617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:28.172183 20617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:28.174314 20617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "1ac31a217637459dbe5f5d17bf772cd6"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "b3ac770098a11121da814704926b0715"
server_key_iv: "3076b1b3e4d32156a361dd0953ef7ef7"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.174875 20617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "1ac31a217637459dbe5f5d17bf772cd6"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "b3ac770098a11121da814704926b0715"
server_key_iv: "3076b1b3e4d32156a361dd0953ef7ef7"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.178817 20617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:01:28.181543 20652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:28.182845 20617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:28.182973 20617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "1ac31a217637459dbe5f5d17bf772cd6"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "b3ac770098a11121da814704926b0715"
server_key_iv: "3076b1b3e4d32156a361dd0953ef7ef7"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.183079 20617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:28.222260 20617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:28.225817 20617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:28.226020 20617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:28.234078 20617 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:42137
I20260810 23:01:28.234405 20745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:42137 every 8 connection(s)
I20260810 23:01:28.235146 20617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:28.238286 20749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:28.240824 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 20617
I20260810 23:01:28.240947 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:28.241250 15584 external_mini_cluster.cc:1428] Setting key 99865d2ab28b3b0bf0ab6d2eb8412d3f
I20260810 23:01:28.245877 20749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6: Bootstrap starting.
I20260810 23:01:28.249120 20749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:28.250118 20749 log.cc:824] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:01:28 dist-test-slave-6bbx krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:28.252593 20749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6: No bootstrap required, opened a new log
I20260810 23:01:28.256302 20749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac31a217637459dbe5f5d17bf772cd6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42137 } }
I20260810 23:01:28.256595 20749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:28.256656 20749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac31a217637459dbe5f5d17bf772cd6, State: Initialized, Role: FOLLOWER
I20260810 23:01:28.257138 20749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac31a217637459dbe5f5d17bf772cd6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42137 } }
I20260810 23:01:28.257300 20749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:28.257369 20749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:28.257473 20749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:28.257648 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:28.243106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.15.56.62:42137)
0810 23:01:28.243647 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:28.243694 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:28.243952 (+ 258us) server_negotiation.cc:407] Connection header received
0810 23:01:28.244715 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:28.244736 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:28.245127 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:28.245463 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:28.246535 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:28.247788 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:28.249007 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:28.249600 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:28.249730 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:28.252644 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:28.252650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:28.252676 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:28.252711 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:28.254430 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:28.254913 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:28.254918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:28.254924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:28.255063 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:28.255362 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:28.255364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:28.255365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:28.255881 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:28.256094 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:28.256401 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 23:01:28.256737 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":166,"threads_started":1}
I20260810 23:01:28.258545 20749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac31a217637459dbe5f5d17bf772cd6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42137 } }
I20260810 23:01:28.258903 20749 leader_election.cc:302] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ac31a217637459dbe5f5d17bf772cd6; no voters:
I20260810 23:01:28.259238 20749 leader_election.cc:288] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:28.259351 20754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:28.259574 20754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1ac31a217637459dbe5f5d17bf772cd6, State: Running, Role: LEADER
I20260810 23:01:28.259931 20754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac31a217637459dbe5f5d17bf772cd6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42137 } }
I20260810 23:01:28.260583 20749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:28.261102 20754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ac31a217637459dbe5f5d17bf772cd6. Latest consensus state: current_term: 1 leader_uuid: "1ac31a217637459dbe5f5d17bf772cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac31a217637459dbe5f5d17bf772cd6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42137 } } }
I20260810 23:01:28.261243 20754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:28.261576 20756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ac31a217637459dbe5f5d17bf772cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac31a217637459dbe5f5d17bf772cd6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42137 } } }
I20260810 23:01:28.261686 20756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:28.261866 20763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:28.264863 20763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:28.270920 20763 catalog_manager.cc:1347] Generated new cluster ID: 4da3b71e752f468e931c24150b6331e4
I20260810 23:01:28.270998 20763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:28.295567 20763 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:28.296710 20763 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:28.301098 20763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ac31a217637459dbe5f5d17bf772cd6: Generated new TSK 0
I20260810 23:01:28.301841 20763 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:01:28.336009 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42137
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:28.454180 20788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:28.454468 20788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:28.454511 20788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:28.458586 20788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:28.458663 20788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:28.458724 20788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:28.464680 20788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:28.465814 20788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:28.466877 20788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:28.477366 20802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:28.477499 20788 server_base.cc:1034] running on GCE node
W20260810 23:01:28.477362 20803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:28.477368 20807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:28.478720 20788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:28.479422 20788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:28.480582 20788 hybrid_clock.cc:648] HybridClock initialized: now 1786402888480569 us; error 45 us; skew 500 ppm
Aug 10 23:01:28 dist-test-slave-6bbx krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:28.483641 20788 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:01:28.484678 20788 webserver.cc:460] Webserver started at http://127.15.56.1:34821/ using document root <none> and password file <none>
I20260810 23:01:28.485308 20788 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:28.485383 20788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:28.485603 20788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:28.487946 20788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "b3d85647efe64dfdb439d0b79908fd78"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "bd79efa2fd2b870502c01cd95863fa24"
server_key_iv: "f9c1a4cbf09766d8640ece4c40976dc1"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.488555 20788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "b3d85647efe64dfdb439d0b79908fd78"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "bd79efa2fd2b870502c01cd95863fa24"
server_key_iv: "f9c1a4cbf09766d8640ece4c40976dc1"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.492574 20788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:01:28.495250 20821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:28.496697 20788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:01:28.496867 20788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "b3d85647efe64dfdb439d0b79908fd78"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "bd79efa2fd2b870502c01cd95863fa24"
server_key_iv: "f9c1a4cbf09766d8640ece4c40976dc1"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.496985 20788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:28.518146 20788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:28.521579 20788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:28.521773 20788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:28.522444 20788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:28.523319 20788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:28.523370 20788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:28.523425 20788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:28.523451 20788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:28.536176 20788 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:33237
I20260810 23:01:28.536180 21031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:33237 every 8 connection(s)
I20260810 23:01:28.537568 20788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:28.546764 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 20788
I20260810 23:01:28.546900 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:28.547252 15584 external_mini_cluster.cc:1428] Setting key 9753c588d701ad2f28ea36f37249d00e
Aug 10 23:01:28 dist-test-slave-6bbx krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:28.553256 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:28.539737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:55509 (local address 127.15.56.62:42137)
0810 23:01:28.539870 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:28.539873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:28.540940 (+ 1067us) server_negotiation.cc:407] Connection header received
0810 23:01:28.542466 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:28.542469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:28.542524 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:28.542593 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:28.544241 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:28.544811 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:28.546012 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:28.546239 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:28.546313 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:28.549147 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:28.549151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:28.549153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:28.549184 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:28.550745 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:28.551501 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:28.551504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:28.551505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:28.551596 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:28.552044 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:28.552047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:28.552048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:28.552252 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:28.552332 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:28.552999 (+ 667us) server_negotiation.cc:299] Negotiation successful
0810 23:01:28.553122 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:01:28.554471 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:28.540169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:42137 (local address 127.15.56.1:55509)
0810 23:01:28.540769 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:28.540820 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:28.542162 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:28.542925 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:28.542935 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:28.543414 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:28.544049 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:28.544067 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:28.544948 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:28.544951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:28.545912 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:28.545923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:28.546365 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:28.546368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:28.546548 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:28.547209 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:28.547228 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:28.548975 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:28.550923 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:28.550930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:28.550952 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:28.551355 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:28.551721 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:28.551724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:28.551726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:28.551943 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:28.552354 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:28.552362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:28.552719 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 23:01:28.552934 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 23:01:28.553319 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":137,"threads_started":1}
I20260810 23:01:28.555816 21032 heartbeater.cc:347] Connected to a master server at 127.15.56.62:42137
I20260810 23:01:28.556079 21032 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:28.556609 21032 heartbeater.cc:511] Master 127.15.56.62:42137 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:01:28.558189 20683 ts_manager.cc:194] Registered new tserver with Master: b3d85647efe64dfdb439d0b79908fd78 (127.15.56.1:33237)
I20260810 23:01:28.559410 20683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:55509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:01:28.576972 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42137
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:28.704413 21042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:28.704701 21042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:28.704746 21042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:28.708802 21042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:28.708868 21042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:28.708928 21042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:28.714185 21042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:28.715294 21042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:28.716382 21042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:28.726387 21050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:28.726478 21042 server_base.cc:1034] running on GCE node
W20260810 23:01:28.726356 21059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:28.726369 21055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:28.727125 21042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:28.727887 21042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:28.729069 21042 hybrid_clock.cc:648] HybridClock initialized: now 1786402888729034 us; error 46 us; skew 500 ppm
Aug 10 23:01:28 dist-test-slave-6bbx krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:28.732137 21042 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:01:28.733217 21042 webserver.cc:460] Webserver started at http://127.15.56.2:39261/ using document root <none> and password file <none>
I20260810 23:01:28.733820 21042 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:28.733894 21042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:28.734120 21042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:28.736567 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "bc6a077a797d43808670bcab4a348552"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "5016f09752d9a95c653e9cb78c1b0b1b"
server_key_iv: "31d0ebc39b11fff7592a0cb69ae5aa84"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.737227 21042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "bc6a077a797d43808670bcab4a348552"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "5016f09752d9a95c653e9cb78c1b0b1b"
server_key_iv: "31d0ebc39b11fff7592a0cb69ae5aa84"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.742013 21042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 23:01:28.744987 21069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:28.746536 21042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:01:28.746692 21042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "bc6a077a797d43808670bcab4a348552"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "5016f09752d9a95c653e9cb78c1b0b1b"
server_key_iv: "31d0ebc39b11fff7592a0cb69ae5aa84"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.746817 21042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:28.769158 21042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:28.773074 21042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:28.773298 21042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:28.774000 21042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:28.774963 21042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:28.775019 21042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:28.775077 21042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:28.775102 21042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:28.784551 21042 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:43269
I20260810 23:01:28.784649 21269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:43269 every 8 connection(s)
I20260810 23:01:28.785539 21042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:28.786374 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 21042
I20260810 23:01:28.786468 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:28.786741 15584 external_mini_cluster.cc:1428] Setting key 7a3cdabd78f383764f14b69da6312131
Aug 10 23:01:28 dist-test-slave-6bbx krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
I20260810 23:01:28.799762 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:28.787574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:41753 (local address 127.15.56.62:42137)
0810 23:01:28.787763 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:28.787768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:28.788849 (+ 1081us) server_negotiation.cc:407] Connection header received
0810 23:01:28.790248 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:28.790252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:28.790308 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:28.790384 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:28.791598 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:28.792181 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:28.793322 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:28.793545 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:28.793634 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:28.796123 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:28.796128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:28.796130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:28.796154 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:28.797533 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:28.798083 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:28.798085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:28.798086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:28.798184 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:28.798594 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:28.798596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:28.798598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:28.798788 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:28.798867 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:28.799466 (+ 599us) server_negotiation.cc:299] Negotiation successful
0810 23:01:28.799587 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:01:28.800861 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:28.788103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:42137 (local address 127.15.56.2:41753)
0810 23:01:28.788687 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:28.788743 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:28.790023 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:28.790570 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:28.790578 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:28.790955 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:28.791452 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:28.791466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:28.792306 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:28.792309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:28.793200 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:28.793210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:28.793692 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:28.793697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:28.793858 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:28.794435 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:28.794452 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:28.795958 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:28.797664 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:28.797669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:28.797683 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:28.798002 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:28.798290 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:28.798293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:28.798294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:28.798503 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:28.798874 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:28.798880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:28.799199 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 23:01:28.799388 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 23:01:28.799752 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":147,"threads_started":1}
I20260810 23:01:28.802134 21270 heartbeater.cc:347] Connected to a master server at 127.15.56.62:42137
I20260810 23:01:28.802389 21270 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:28.802927 21270 heartbeater.cc:511] Master 127.15.56.62:42137 requested a full tablet report, sending...
I20260810 23:01:28.804082 20683 ts_manager.cc:194] Registered new tserver with Master: bc6a077a797d43808670bcab4a348552 (127.15.56.2:43269)
I20260810 23:01:28.804860 20683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:41753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:01:28.815340 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42137
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:28.932796 21284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:28.933110 21284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:28.933164 21284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:28.937299 21284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:28.937362 21284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:28.937422 21284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:28.943378 21284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:42137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:28.944624 21284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:28.945708 21284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:28.955694 21294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:28.955829 21284 server_base.cc:1034] running on GCE node
W20260810 23:01:28.955780 21299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:28.956043 21295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:28.956488 21284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:28.957199 21284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:28.958372 21284 hybrid_clock.cc:648] HybridClock initialized: now 1786402888958337 us; error 44 us; skew 500 ppm
Aug 10 23:01:28 dist-test-slave-6bbx krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:28.961362 21284 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:01:28.962301 21284 webserver.cc:460] Webserver started at http://127.15.56.3:44703/ using document root <none> and password file <none>
I20260810 23:01:28.962869 21284 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:28.962937 21284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:28.963145 21284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:28.965250 21284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "bae2fc472eac4397aecfb794b7f38e99"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "07e3a4737db24451786ead4fdd9fca93"
server_key_iv: "7f2fc4f2afea04ec4e5859fa2f938f21"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.965797 21284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "bae2fc472eac4397aecfb794b7f38e99"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "07e3a4737db24451786ead4fdd9fca93"
server_key_iv: "7f2fc4f2afea04ec4e5859fa2f938f21"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.969538 21284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260810 23:01:28.972085 21311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:28.973526 21284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:28.973634 21284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "bae2fc472eac4397aecfb794b7f38e99"
format_stamp: "Formatted at 2026-08-10 23:01:28 on dist-test-slave-6bbx"
server_key: "07e3a4737db24451786ead4fdd9fca93"
server_key_iv: "7f2fc4f2afea04ec4e5859fa2f938f21"
server_key_version: "encryptionkey@0"
I20260810 23:01:28.973723 21284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:28.987473 21284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:28.990796 21284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:28.990965 21284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:28.991557 21284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:28.992466 21284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:28.992516 21284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:28.992558 21284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:28.992576 21284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:29.001588 21284 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:43199
I20260810 23:01:29.001689 21511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:43199 every 8 connection(s)
I20260810 23:01:29.002642 21284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:29.012462 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 21284
I20260810 23:01:29.012609 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:29.012929 15584 external_mini_cluster.cc:1428] Setting key 2dc98e5957986e7b52448765f7b5e0b9
Aug 10 23:01:29 dist-test-slave-6bbx krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402888, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:29.017552 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:29.004743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:43949 (local address 127.15.56.62:42137)
0810 23:01:29.004900 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:29.004905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:29.005904 (+ 999us) server_negotiation.cc:407] Connection header received
0810 23:01:29.007275 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:29.007277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:29.007356 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:29.007426 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:29.008770 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:29.009322 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:29.010580 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:29.010826 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:29.010923 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:29.013904 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:29.013907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:29.013909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:29.013937 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:29.015338 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:29.015943 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:29.015946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:29.015947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:29.016031 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:29.016379 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:29.016382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:29.016383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:29.016581 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:29.016670 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:29.017239 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 23:01:29.017387 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:01:29.018445 21517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:29.005203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:42137 (local address 127.15.56.3:43949)
0810 23:01:29.005730 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:29.005778 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:29.007035 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:29.007597 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:29.007605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:29.008076 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:29.008607 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:29.008622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:29.009481 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:29.009484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:29.010428 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:29.010439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:29.011039 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:29.011042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:29.011248 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:29.011919 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:29.011938 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:29.013692 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:29.015460 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:29.015467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:29.015487 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:29.015843 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:29.016119 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:29.016122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:29.016124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:29.016278 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:29.016686 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:29.016693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:29.016979 (+ 286us) client_negotiation.cc:769] Sending connection context
0810 23:01:29.017157 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 23:01:29.017553 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":133,"threads_started":1}
I20260810 23:01:29.019591 21513 heartbeater.cc:347] Connected to a master server at 127.15.56.62:42137
I20260810 23:01:29.019891 21513 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:29.020434 21513 heartbeater.cc:511] Master 127.15.56.62:42137 requested a full tablet report, sending...
I20260810 23:01:29.021441 20683 ts_manager.cc:194] Registered new tserver with Master: bae2fc472eac4397aecfb794b7f38e99 (127.15.56.3:43199)
I20260810 23:01:29.022019 20683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:43949
I20260810 23:01:29.026890 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:01:29 dist-test-slave-6bbx krb5kdc[20596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402889, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 23:01:29 dist-test-slave-6bbx krb5kdc[20596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402889, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:29.054441 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:29.045789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.15.56.62:42137)
0810 23:01:29.045914 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:29.045917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:29.045999 (+ 82us) server_negotiation.cc:407] Connection header received
0810 23:01:29.046113 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:29.046115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:29.046166 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:29.046249 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:29.047011 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:29.047551 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:29.048444 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:29.048694 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:29.048785 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:29.051553 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:29.051557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:29.051559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:29.051588 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:29.052828 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:29.053266 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:29.053268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:29.053269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:29.053344 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:29.053617 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:29.053620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:29.053621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:29.053838 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:29.053945 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:29.054200 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 23:01:29.054310 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 23:01:29.054896 20683 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58342
I20260810 23:01:29.055792 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 20788
I20260810 23:01:29.074954 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 21042
I20260810 23:01:29.095201 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 21284
I20260810 23:01:29.123078 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 20617
2026-08-10T23:01:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2397 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:29 dist-test-slave-6bbx krb5kdc[21534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:29 dist-test-slave-6bbx krb5kdc[21534](info): set up 2 sockets
Aug 10 23:01:29 dist-test-slave-6bbx krb5kdc[21534](info): commencing operation
krb5kdc: starting...
W20260810 23:01:30.286696 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:30 dist-test-slave-6bbx krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:30.408416 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:46723
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:46723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:30.519657 21559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:30.519953 21559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:30.520007 21559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:30.523996 21559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:30.524048 21559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:30.524075 21559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:30.524096 21559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:30.524124 21559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:30.529284 21559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:46723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:46723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:30.530278 21559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:30.531273 21559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:30.539763 21570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:30.539763 21568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:30.539796 21578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:30.539948 21559 server_base.cc:1034] running on GCE node
I20260810 23:01:30.540647 21559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:30.541755 21559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:30.542927 21559 hybrid_clock.cc:648] HybridClock initialized: now 1786402890542911 us; error 31 us; skew 500 ppm
Aug 10 23:01:30 dist-test-slave-6bbx krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402890, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:30.550423 21559 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:30.551385 21559 webserver.cc:460] Webserver started at http://127.15.56.62:38129/ using document root <none> and password file <none>
I20260810 23:01:30.551982 21559 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:30.552048 21559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:30.552234 21559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:30.554292 21559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "0caace4a98cf4ff187889b98b882b7be"
format_stamp: "Formatted at 2026-08-10 23:01:30 on dist-test-slave-6bbx"
server_key: "7e87c9e4c5be3766a616649aa3a49ebc"
server_key_iv: "a92e6be5a99d2d8af19d0ffdeb37099a"
server_key_version: "encryptionkey@0"
I20260810 23:01:30.554823 21559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "0caace4a98cf4ff187889b98b882b7be"
format_stamp: "Formatted at 2026-08-10 23:01:30 on dist-test-slave-6bbx"
server_key: "7e87c9e4c5be3766a616649aa3a49ebc"
server_key_iv: "a92e6be5a99d2d8af19d0ffdeb37099a"
server_key_version: "encryptionkey@0"
I20260810 23:01:30.558604 21559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:01:30.560997 21586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:30.562225 21559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:01:30.562333 21559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "0caace4a98cf4ff187889b98b882b7be"
format_stamp: "Formatted at 2026-08-10 23:01:30 on dist-test-slave-6bbx"
server_key: "7e87c9e4c5be3766a616649aa3a49ebc"
server_key_iv: "a92e6be5a99d2d8af19d0ffdeb37099a"
server_key_version: "encryptionkey@0"
I20260810 23:01:30.562430 21559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:30.577636 21559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:30.580703 21559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:30.580883 21559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:30.588119 21559 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:46723
I20260810 23:01:30.588183 21676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:46723 every 8 connection(s)
I20260810 23:01:30.589094 21559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:30.592000 21678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:30.594906 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 21559
I20260810 23:01:30.595029 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:30.595324 15584 external_mini_cluster.cc:1428] Setting key 54ade3ceef941d4c8c3c4eb0898eb496
I20260810 23:01:30.598043 21678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be: Bootstrap starting.
I20260810 23:01:30.600579 21678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:30.601470 21678 log.cc:824] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:30.603826 21678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be: No bootstrap required, opened a new log
Aug 10 23:01:30 dist-test-slave-6bbx krb5kdc[21534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:30.607045 21678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caace4a98cf4ff187889b98b882b7be" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46723 } }
I20260810 23:01:30.607283 21678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:30.607339 21678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0caace4a98cf4ff187889b98b882b7be, State: Initialized, Role: FOLLOWER
I20260810 23:01:30.607774 21678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caace4a98cf4ff187889b98b882b7be" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46723 } }
I20260810 23:01:30.607918 21678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:30.607988 21678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:30.608098 21678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:30.609083 21678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caace4a98cf4ff187889b98b882b7be" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46723 } }
I20260810 23:01:30.609448 21678 leader_election.cc:302] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0caace4a98cf4ff187889b98b882b7be; no voters:
I20260810 23:01:30.609768 21678 leader_election.cc:288] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:30.609855 21684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:30.610025 21684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [term 1 LEADER]: Becoming Leader. State: Replica: 0caace4a98cf4ff187889b98b882b7be, State: Running, Role: LEADER
I20260810 23:01:30.610285 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:30.596716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.15.56.62:46723)
0810 23:01:30.597110 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:30.597126 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:30.597163 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:01:30.597773 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:30.597791 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:30.598121 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:30.598422 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:30.599330 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:30.600396 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:30.601551 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:30.602217 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:30.602359 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:30.605026 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:30.605031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:30.605046 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:30.605073 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:30.607225 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:30.607743 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:30.607748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:30.607753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:30.607892 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:30.608218 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:30.608221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:30.608223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:30.608621 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:30.608795 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:30.609081 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 23:01:30.609374 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260810 23:01:30.610319 21684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caace4a98cf4ff187889b98b882b7be" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46723 } }
I20260810 23:01:30.611047 21678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:30.612217 21687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0caace4a98cf4ff187889b98b882b7be. Latest consensus state: current_term: 1 leader_uuid: "0caace4a98cf4ff187889b98b882b7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caace4a98cf4ff187889b98b882b7be" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46723 } } }
I20260810 23:01:30.612223 21688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0caace4a98cf4ff187889b98b882b7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caace4a98cf4ff187889b98b882b7be" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46723 } } }
I20260810 23:01:30.612347 21687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:30.612351 21688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be [sys.catalog]: This master's current role is: LEADER
W20260810 23:01:30.615775 21707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:01:30.615841 21707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:01:30.615922 21700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:30.616596 21700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:30.621703 21700 catalog_manager.cc:1347] Generated new cluster ID: 328e25c6dcd24252965dc75ad2dcb7ee
I20260810 23:01:30.621768 21700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:30.645612 21700 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:30.646589 21700 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:30.650422 21700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0caace4a98cf4ff187889b98b882b7be: Generated new TSK 0
I20260810 23:01:30.651153 21700 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:01:30.686765 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46723
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:30.803411 21718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:30.803727 21718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:30.803774 21718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:30.807884 21718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:30.807955 21718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:30.808022 21718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:30.813687 21718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:30.814858 21718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:30.815963 21718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:30.826313 21727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:30.826331 21729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:30.826305 21733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:30.826650 21718 server_base.cc:1034] running on GCE node
I20260810 23:01:30.827090 21718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:30.827775 21718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:30.828954 21718 hybrid_clock.cc:648] HybridClock initialized: now 1786402890828917 us; error 47 us; skew 500 ppm
Aug 10 23:01:30 dist-test-slave-6bbx krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402890, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:30.831990 21718 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:01:30.832980 21718 webserver.cc:460] Webserver started at http://127.15.56.1:43445/ using document root <none> and password file <none>
I20260810 23:01:30.833566 21718 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:30.833633 21718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:30.833856 21718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:30.836023 21718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "e2de1c1fa0c64ace99cf0aa84f7e20cd"
format_stamp: "Formatted at 2026-08-10 23:01:30 on dist-test-slave-6bbx"
server_key: "682ee207f1e86682823458098db0b904"
server_key_iv: "86fa1f08ba14718782c08fb46b2f22e4"
server_key_version: "encryptionkey@0"
I20260810 23:01:30.836588 21718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "e2de1c1fa0c64ace99cf0aa84f7e20cd"
format_stamp: "Formatted at 2026-08-10 23:01:30 on dist-test-slave-6bbx"
server_key: "682ee207f1e86682823458098db0b904"
server_key_iv: "86fa1f08ba14718782c08fb46b2f22e4"
server_key_version: "encryptionkey@0"
I20260810 23:01:30.840405 21718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:01:30.842953 21742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:30.844316 21718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:30.844435 21718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "e2de1c1fa0c64ace99cf0aa84f7e20cd"
format_stamp: "Formatted at 2026-08-10 23:01:30 on dist-test-slave-6bbx"
server_key: "682ee207f1e86682823458098db0b904"
server_key_iv: "86fa1f08ba14718782c08fb46b2f22e4"
server_key_version: "encryptionkey@0"
I20260810 23:01:30.844532 21718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:30.861999 21718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:30.865423 21718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:30.865608 21718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:30.866262 21718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:30.867240 21718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:30.867316 21718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:30.867377 21718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:30.867403 21718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:30.875850 21718 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:32921
I20260810 23:01:30.875906 21954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:32921 every 8 connection(s)
I20260810 23:01:30.876804 21718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:30.883921 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 21718
I20260810 23:01:30.884038 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:30.884352 15584 external_mini_cluster.cc:1428] Setting key 4204c82ddbc24ca8a81e7223a79a932e
Aug 10 23:01:30 dist-test-slave-6bbx krb5kdc[21534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402890, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:30.890892 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:30.878613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:46609 (local address 127.15.56.62:46723)
0810 23:01:30.878759 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:30.878762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:30.879617 (+ 855us) server_negotiation.cc:407] Connection header received
0810 23:01:30.881023 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:30.881025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:30.881073 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:30.881153 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:30.882341 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:30.882878 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:30.884130 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:30.884458 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:30.884571 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:30.887439 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:30.887443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:30.887445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:30.887470 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:30.888871 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:30.889377 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:30.889380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:30.889382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:30.889466 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:30.889800 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:30.889803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:30.889805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:30.889991 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:30.890078 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:30.890589 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 23:01:30.890722 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:01:30.891793 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:30.878990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:46723 (local address 127.15.56.1:46609)
0810 23:01:30.879458 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:30.879503 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:30.880817 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:30.881309 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:30.881316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:30.881687 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:30.882194 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:30.882211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:30.883017 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:30.883021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:30.883993 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:30.884003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:30.884609 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:30.884613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:30.884781 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:30.885368 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:30.885385 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:30.887241 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:30.888990 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:30.888996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:30.889010 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:30.889279 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:30.889552 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:30.889555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:30.889557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:30.889695 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:30.890093 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:30.890099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:30.890355 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 23:01:30.890529 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 23:01:30.890887 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260810 23:01:30.893019 21956 heartbeater.cc:347] Connected to a master server at 127.15.56.62:46723
I20260810 23:01:30.893301 21956 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:30.893930 21956 heartbeater.cc:511] Master 127.15.56.62:46723 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:01:30.895601 21623 ts_manager.cc:194] Registered new tserver with Master: e2de1c1fa0c64ace99cf0aa84f7e20cd (127.15.56.1:32921)
I20260810 23:01:30.896767 21623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:46609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:01:30.912298 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46723
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:31.027129 21966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:31.027400 21966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:31.027444 21966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:31.031471 21966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:31.031541 21966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:31.031600 21966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:31.037042 21966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:31.038211 21966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:31.039294 21966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:31.049429 21983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:31.049433 21973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:31.049580 21966 server_base.cc:1034] running on GCE node
W20260810 23:01:31.049433 21976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:31.050189 21966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:31.050897 21966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:31.052066 21966 hybrid_clock.cc:648] HybridClock initialized: now 1786402891052044 us; error 34 us; skew 500 ppm
Aug 10 23:01:31 dist-test-slave-6bbx krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402891, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:31.054913 21966 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:01:31.055873 21966 webserver.cc:460] Webserver started at http://127.15.56.2:43323/ using document root <none> and password file <none>
I20260810 23:01:31.056473 21966 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:31.056538 21966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:31.056712 21966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:31.058960 21966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "ba3758c1891c46ea8574e478c903f4f6"
format_stamp: "Formatted at 2026-08-10 23:01:31 on dist-test-slave-6bbx"
server_key: "2559764d99bfd27c801e661a3fb885b5"
server_key_iv: "a6171cd6f98b53bddaa16a4dce26f778"
server_key_version: "encryptionkey@0"
I20260810 23:01:31.059554 21966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "ba3758c1891c46ea8574e478c903f4f6"
format_stamp: "Formatted at 2026-08-10 23:01:31 on dist-test-slave-6bbx"
server_key: "2559764d99bfd27c801e661a3fb885b5"
server_key_iv: "a6171cd6f98b53bddaa16a4dce26f778"
server_key_version: "encryptionkey@0"
I20260810 23:01:31.063647 21966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 23:01:31.066406 21997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:31.067894 21966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:01:31.068009 21966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "ba3758c1891c46ea8574e478c903f4f6"
format_stamp: "Formatted at 2026-08-10 23:01:31 on dist-test-slave-6bbx"
server_key: "2559764d99bfd27c801e661a3fb885b5"
server_key_iv: "a6171cd6f98b53bddaa16a4dce26f778"
server_key_version: "encryptionkey@0"
I20260810 23:01:31.068109 21966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:31.082335 21966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:31.085372 21966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:31.085556 21966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:31.086151 21966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:31.086999 21966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:31.087049 21966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:31.087105 21966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:31.087134 21966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:31.095758 21966 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:45113
I20260810 23:01:31.095781 22194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:45113 every 8 connection(s)
I20260810 23:01:31.096716 21966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:31.099406 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 21966
I20260810 23:01:31.099558 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:31.099956 15584 external_mini_cluster.cc:1428] Setting key 0f735c67b395f856aa344c301592af9f
Aug 10 23:01:31 dist-test-slave-6bbx krb5kdc[21534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402891, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:01:31.112237 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:31.098664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:56667 (local address 127.15.56.62:46723)
0810 23:01:31.098907 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:31.098911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:31.099918 (+ 1007us) server_negotiation.cc:407] Connection header received
0810 23:01:31.101493 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:31.101497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:31.101545 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:31.101603 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:31.103228 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.103833 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.105427 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.105645 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.105738 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:31.108538 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:31.108541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:31.108543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:31.108568 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:31.110005 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:31.110667 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:31.110671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:31.110673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:31.110761 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:31.111121 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:31.111123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:31.111124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:31.111326 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:31.111401 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:31.111991 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 23:01:31.112109 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:01:31.113148 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:31.099087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:46723 (local address 127.15.56.2:56667)
0810 23:01:31.099714 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:31.099785 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:31.101269 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:31.101822 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:31.101832 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:31.102377 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:31.103067 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:31.103088 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.103960 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.103964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:31.105300 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:31.105311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.105807 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.105812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:31.105993 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:31.106585 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:31.106602 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:31.108373 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:31.110122 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:31.110129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:31.110154 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:31.110512 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:31.110857 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:31.110860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:31.110862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:31.111038 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:31.111430 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:31.111436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:31.111720 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 23:01:31.111925 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 23:01:31.112266 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
I20260810 23:01:31.114491 22201 heartbeater.cc:347] Connected to a master server at 127.15.56.62:46723
I20260810 23:01:31.114753 22201 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:31.115271 22201 heartbeater.cc:511] Master 127.15.56.62:46723 requested a full tablet report, sending...
I20260810 23:01:31.116460 21623 ts_manager.cc:194] Registered new tserver with Master: ba3758c1891c46ea8574e478c903f4f6 (127.15.56.2:45113)
I20260810 23:01:31.117024 21623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:56667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:01:31.128928 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46723
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:31.242084 22210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:31.242378 22210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:31.242420 22210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:31.246663 22210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:31.246731 22210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:31.246791 22210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:31.253068 22210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:31.254297 22210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:31.255409 22210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:31.266036 22219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:31.266078 22210 server_base.cc:1034] running on GCE node
W20260810 23:01:31.266036 22223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:31.266394 22220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:31.266886 22210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:31.267593 22210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:31.268821 22210 hybrid_clock.cc:648] HybridClock initialized: now 1786402891268798 us; error 33 us; skew 500 ppm
Aug 10 23:01:31 dist-test-slave-6bbx krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402891, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:31.272154 22210 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:01:31.273286 22210 webserver.cc:460] Webserver started at http://127.15.56.3:36059/ using document root <none> and password file <none>
I20260810 23:01:31.273916 22210 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:31.273996 22210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:31.274217 22210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:31.276607 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "9b561ce299dc49b69a0580ab7d28cc27"
format_stamp: "Formatted at 2026-08-10 23:01:31 on dist-test-slave-6bbx"
server_key: "fc96830c15d492524ccb9cbdc3993963"
server_key_iv: "c815b652f72f2b2b8668b008219913c3"
server_key_version: "encryptionkey@0"
I20260810 23:01:31.277251 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "9b561ce299dc49b69a0580ab7d28cc27"
format_stamp: "Formatted at 2026-08-10 23:01:31 on dist-test-slave-6bbx"
server_key: "fc96830c15d492524ccb9cbdc3993963"
server_key_iv: "c815b652f72f2b2b8668b008219913c3"
server_key_version: "encryptionkey@0"
I20260810 23:01:31.281476 22210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:01:31.284175 22236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:31.285545 22210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:31.285672 22210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "9b561ce299dc49b69a0580ab7d28cc27"
format_stamp: "Formatted at 2026-08-10 23:01:31 on dist-test-slave-6bbx"
server_key: "fc96830c15d492524ccb9cbdc3993963"
server_key_iv: "c815b652f72f2b2b8668b008219913c3"
server_key_version: "encryptionkey@0"
I20260810 23:01:31.285789 22210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:31.298950 22210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:31.302243 22210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:31.302443 22210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:31.303073 22210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:31.304008 22210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:31.304061 22210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:31.304118 22210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:31.304144 22210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:31.313112 22210 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:33409
I20260810 23:01:31.313155 22472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:33409 every 8 connection(s)
I20260810 23:01:31.314107 22210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:31.316282 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 22210
I20260810 23:01:31.316427 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:31.316711 15584 external_mini_cluster.cc:1428] Setting key d6bca9263ffeb87866e1b697e9b31349
Aug 10 23:01:31 dist-test-slave-6bbx krb5kdc[21534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402891, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:31.328912 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:31.316129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:44613 (local address 127.15.56.62:46723)
0810 23:01:31.316306 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:31.316311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:31.317259 (+ 948us) server_negotiation.cc:407] Connection header received
0810 23:01:31.318970 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:31.318973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:31.319029 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:31.319110 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:31.320389 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.320930 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.322148 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.322391 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.322483 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:31.325232 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:31.325237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:31.325240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:31.325270 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:31.326629 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:31.327195 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:31.327199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:31.327201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:31.327291 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:31.327650 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:31.327701 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:31.327702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:31.327921 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:31.328016 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:31.328580 (+ 564us) server_negotiation.cc:299] Negotiation successful
0810 23:01:31.328739 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:01:31.329977 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:31.316576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:46723 (local address 127.15.56.3:44613)
0810 23:01:31.317107 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:31.317156 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:31.318722 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:31.319271 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:31.319280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:31.319659 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:31.320256 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:31.320272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.321055 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.321058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:31.321997 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:31.322007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.322533 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.322536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:31.322691 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:31.323396 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:31.323414 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:31.325033 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:31.326778 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:31.326785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:31.326801 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:31.327087 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:31.327390 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:31.327393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:31.327396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:31.327554 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:31.328032 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:31.328040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:31.328330 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 23:01:31.328520 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 23:01:31.328906 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":133,"threads_started":1}
I20260810 23:01:31.331246 22473 heartbeater.cc:347] Connected to a master server at 127.15.56.62:46723
I20260810 23:01:31.331523 22473 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:31.332122 22473 heartbeater.cc:511] Master 127.15.56.62:46723 requested a full tablet report, sending...
I20260810 23:01:31.333285 21617 ts_manager.cc:194] Registered new tserver with Master: 9b561ce299dc49b69a0580ab7d28cc27 (127.15.56.3:33409)
I20260810 23:01:31.333920 21617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:44613
I20260810 23:01:31.343230 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:01:31 dist-test-slave-6bbx krb5kdc[21534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:01:31 dist-test-slave-6bbx krb5kdc[21534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:01:31.371169 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:31.361451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.15.56.1:32921)
0810 23:01:31.361780 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:31.361796 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:31.361822 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:01:31.361878 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:31.361884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:31.362092 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:31.362271 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:31.362929 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.363825 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.364806 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.365203 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.365293 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:31.367789 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:31.367799 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:31.367814 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:31.367840 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:31.369441 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:31.369859 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:31.369865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:31.369866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:31.369951 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:31.370140 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:31.370143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:31.370167 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:31.370414 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:31.370603 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:31.370818 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 23:01:31.370957 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
Aug 10 23:01:31 dist-test-slave-6bbx krb5kdc[21534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:31.382217 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:31.374307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.15.56.62:46723)
0810 23:01:31.374416 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:31.374419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:31.374432 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:01:31.374540 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:31.374542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:31.374582 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:31.374646 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:31.375199 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.375754 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.376579 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:31.376821 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:31.376903 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:31.379267 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:31.379271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:31.379273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:31.379299 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:31.380627 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:31.381011 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:31.381013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:31.381015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:31.381093 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:31.381343 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:31.381346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:31.381348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:31.381573 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:31.381703 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:31.381926 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 23:01:31.382044 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260810 23:01:31.382620 21617 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45766
I20260810 23:01:31.383255 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 21718
I20260810 23:01:31.412254 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 21966
I20260810 23:01:31.435119 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 22210
I20260810 23:01:31.454947 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 21559
2026-08-10T23:01:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2330 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:31 dist-test-slave-6bbx krb5kdc[22498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:31 dist-test-slave-6bbx krb5kdc[22498](info): set up 2 sockets
Aug 10 23:01:31 dist-test-slave-6bbx krb5kdc[22498](info): commencing operation
krb5kdc: starting...
W20260810 23:01:32.648923 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:32 dist-test-slave-6bbx krb5kdc[22498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:32.774459 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33899
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:33899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:32.888041 22520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:32.888331 22520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:32.888393 22520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:32.892417 22520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:32.892486 22520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:32.892514 22520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:32.892536 22520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:32.892565 22520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:32.897825 22520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:33899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:32.898900 22520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:32.899999 22520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:32.908437 22539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:32.908445 22532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:32.908439 22534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:32.908499 22520 server_base.cc:1034] running on GCE node
I20260810 23:01:32.909332 22520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:32.910554 22520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:32.911715 22520 hybrid_clock.cc:648] HybridClock initialized: now 1786402892911687 us; error 47 us; skew 500 ppm
Aug 10 23:01:32 dist-test-slave-6bbx krb5kdc[22498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402892, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:32.914505 22520 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:32.915243 22520 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:32.915323 22520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:32.915539 22520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:32.917948 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "f3c92d9297244a26b05949262c13eb03"
format_stamp: "Formatted at 2026-08-10 23:01:32 on dist-test-slave-6bbx"
server_key: "062b8fd290391701c818687458cc5b01"
server_key_iv: "40004b9590fdc69a0bfd2ed94a4ffc36"
server_key_version: "encryptionkey@0"
I20260810 23:01:32.918542 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "f3c92d9297244a26b05949262c13eb03"
format_stamp: "Formatted at 2026-08-10 23:01:32 on dist-test-slave-6bbx"
server_key: "062b8fd290391701c818687458cc5b01"
server_key_iv: "40004b9590fdc69a0bfd2ed94a4ffc36"
server_key_version: "encryptionkey@0"
I20260810 23:01:32.922470 22520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 23:01:32.924986 22549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:32.926261 22520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:01:32.926388 22520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "f3c92d9297244a26b05949262c13eb03"
format_stamp: "Formatted at 2026-08-10 23:01:32 on dist-test-slave-6bbx"
server_key: "062b8fd290391701c818687458cc5b01"
server_key_iv: "40004b9590fdc69a0bfd2ed94a4ffc36"
server_key_version: "encryptionkey@0"
I20260810 23:01:32.926483 22520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:32.938454 22520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:32.941555 22520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:32.941736 22520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:32.948475 22520 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:33899
I20260810 23:01:32.948715 22636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:33899 every 8 connection(s)
I20260810 23:01:32.949136 22520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:32.950736 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 22520
I20260810 23:01:32.950855 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:32.951122 15584 external_mini_cluster.cc:1428] Setting key 2c01a5f8ba133d2be232425e72e6712b
I20260810 23:01:32.952219 22637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:32.958832 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03: Bootstrap starting.
Aug 10 23:01:32 dist-test-slave-6bbx krb5kdc[22498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:32.961190 22637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:32.962134 22637 log.cc:824] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:32.964455 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03: No bootstrap required, opened a new log
I20260810 23:01:32.965924 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:32.952537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.15.56.62:33899)
0810 23:01:32.952951 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:32.952971 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:32.953014 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:01:32.953847 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:32.953871 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:32.954301 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:32.954678 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:32.955379 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:32.956875 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:32.957733 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:32.958412 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:32.958568 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:32.960877 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:32.960883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:32.960899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:32.960931 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:32.962797 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:32.963207 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:32.963213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:32.963219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:32.963360 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:32.963586 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:32.963589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:32.963591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:32.964117 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:32.964330 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:32.964515 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 23:01:32.964982 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":122,"threads_started":1}
I20260810 23:01:32.969030 22637 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c92d9297244a26b05949262c13eb03" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33899 } }
I20260810 23:01:32.969260 22637 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:32.969308 22637 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c92d9297244a26b05949262c13eb03, State: Initialized, Role: FOLLOWER
I20260810 23:01:32.969725 22637 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c92d9297244a26b05949262c13eb03" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33899 } }
I20260810 23:01:32.969846 22637 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:32.969902 22637 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:32.969998 22637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:32.970924 22637 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c92d9297244a26b05949262c13eb03" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33899 } }
I20260810 23:01:32.971247 22637 leader_election.cc:302] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3c92d9297244a26b05949262c13eb03; no voters:
I20260810 23:01:32.971558 22637 leader_election.cc:288] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:32.971637 22645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:32.971879 22645 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [term 1 LEADER]: Becoming Leader. State: Replica: f3c92d9297244a26b05949262c13eb03, State: Running, Role: LEADER
I20260810 23:01:32.972180 22645 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c92d9297244a26b05949262c13eb03" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33899 } }
I20260810 23:01:32.972853 22637 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:32.973845 22646 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3c92d9297244a26b05949262c13eb03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c92d9297244a26b05949262c13eb03" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33899 } } }
I20260810 23:01:32.973980 22646 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:32.974437 22648 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3c92d9297244a26b05949262c13eb03. Latest consensus state: current_term: 1 leader_uuid: "f3c92d9297244a26b05949262c13eb03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3c92d9297244a26b05949262c13eb03" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33899 } } }
I20260810 23:01:32.974548 22648 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03 [sys.catalog]: This master's current role is: LEADER
W20260810 23:01:32.977766 22671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:01:32.977843 22671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:01:32.977916 22659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:32.978601 22659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:32.984005 22659 catalog_manager.cc:1347] Generated new cluster ID: 07e1373b6d784160929568050b44ed89
I20260810 23:01:32.984073 22659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:32.997577 22659 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:32.998534 22659 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:33.004465 22659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03: Generated new TSK 0
I20260810 23:01:33.005169 22659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:01:33.038775 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33899
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:33.158674 22685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:33.158943 22685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:33.159005 22685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:33.163082 22685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:33.163147 22685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:33.163237 22685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:33.168555 22685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:33.169636 22685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:33.170691 22685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:33.180424 22697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:33.180416 22699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:33.180478 22685 server_base.cc:1034] running on GCE node
W20260810 23:01:33.180419 22694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:33.181133 22685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:33.181757 22685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:33.182910 22685 hybrid_clock.cc:648] HybridClock initialized: now 1786402893182880 us; error 43 us; skew 500 ppm
Aug 10 23:01:33 dist-test-slave-6bbx krb5kdc[22498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402893, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:33.185792 22685 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:01:33.186568 22685 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:33.186638 22685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:33.186859 22685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:33.188997 22685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "0ade91a074eb45e5b32e92d4e5d45417"
format_stamp: "Formatted at 2026-08-10 23:01:33 on dist-test-slave-6bbx"
server_key: "fbd6ac10644d48d5c293e9b1a95d1582"
server_key_iv: "1d5433321034260c4a5cd00e956b7ded"
server_key_version: "encryptionkey@0"
I20260810 23:01:33.189561 22685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "0ade91a074eb45e5b32e92d4e5d45417"
format_stamp: "Formatted at 2026-08-10 23:01:33 on dist-test-slave-6bbx"
server_key: "fbd6ac10644d48d5c293e9b1a95d1582"
server_key_iv: "1d5433321034260c4a5cd00e956b7ded"
server_key_version: "encryptionkey@0"
I20260810 23:01:33.193447 22685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 23:01:33.196240 22715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:33.197705 22685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:33.197844 22685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "0ade91a074eb45e5b32e92d4e5d45417"
format_stamp: "Formatted at 2026-08-10 23:01:33 on dist-test-slave-6bbx"
server_key: "fbd6ac10644d48d5c293e9b1a95d1582"
server_key_iv: "1d5433321034260c4a5cd00e956b7ded"
server_key_version: "encryptionkey@0"
I20260810 23:01:33.197970 22685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:33.224766 22685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:33.227905 22685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:33.228083 22685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:33.228688 22685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:33.229550 22685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:33.229604 22685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:33.229645 22685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:33.229658 22685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:33.238200 22685 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:33417
I20260810 23:01:33.238240 22932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:33417 every 8 connection(s)
I20260810 23:01:33.238808 22685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:33.246528 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 22685
I20260810 23:01:33.246640 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:33.246923 15584 external_mini_cluster.cc:1428] Setting key d1fc863a4e6762ffe8b9c39b83773fa8
Aug 10 23:01:33 dist-test-slave-6bbx krb5kdc[22498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402893, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:33.251852 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.240593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:56129 (local address 127.15.56.62:33899)
0810 23:01:33.240748 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:33.240751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:33.241766 (+ 1015us) server_negotiation.cc:407] Connection header received
0810 23:01:33.242630 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:33.242633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:33.242679 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:33.242748 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:33.243943 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.244488 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.245571 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.245785 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.245853 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.248424 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:33.248428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:33.248430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:33.248458 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:33.249776 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:33.250256 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:33.250260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:33.250262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:33.250365 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:33.250691 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:33.250694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:33.250695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:33.250893 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:33.250983 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:33.251494 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 23:01:33.251635 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:01:33.252656 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.240975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33899 (local address 127.15.56.1:56129)
0810 23:01:33.241626 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:33.241684 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:33.242454 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:33.242902 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:33.242909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:33.243265 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:33.243804 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.243821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.244586 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.244589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.245480 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.245490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.245894 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.245896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.246039 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.246618 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:33.246640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:33.248276 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:33.249885 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:33.249891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:33.249907 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:33.250162 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:33.250452 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:33.250455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:33.250457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:33.250601 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:33.250994 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:33.251002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:33.251255 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 23:01:33.251432 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 23:01:33.251810 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":115,"threads_started":1}
I20260810 23:01:33.254101 22933 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33899
I20260810 23:01:33.254333 22933 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:33.254809 22933 heartbeater.cc:511] Master 127.15.56.62:33899 requested a full tablet report, sending...
I20260810 23:01:33.256629 22580 ts_manager.cc:194] Registered new tserver with Master: 0ade91a074eb45e5b32e92d4e5d45417 (127.15.56.1:33417)
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:01:33.258249 22580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:56129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:01:33.276471 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33899
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:33.396373 22941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:33.396684 22941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:33.396730 22941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:33.400914 22941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:33.401000 22941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:33.401068 22941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:33.406840 22941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:33.408115 22941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:33.409252 22941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:33.419086 22950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:33.419086 22953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:33.419086 22949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:33.419368 22941 server_base.cc:1034] running on GCE node
I20260810 23:01:33.419922 22941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:33.420622 22941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:33.421860 22941 hybrid_clock.cc:648] HybridClock initialized: now 1786402893421834 us; error 37 us; skew 500 ppm
Aug 10 23:01:33 dist-test-slave-6bbx krb5kdc[22498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402893, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:33.425247 22941 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:01:33.426017 22941 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:33.426085 22941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:33.426275 22941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:33.428516 22941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "fdf61435f1704fb08d4be117ae2b4bd0"
format_stamp: "Formatted at 2026-08-10 23:01:33 on dist-test-slave-6bbx"
server_key: "90cbf3f0f4cb710ff8e2165161e41467"
server_key_iv: "ab07c13a0bcc9ca0c31d646b40fc43b1"
server_key_version: "encryptionkey@0"
I20260810 23:01:33.429083 22941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "fdf61435f1704fb08d4be117ae2b4bd0"
format_stamp: "Formatted at 2026-08-10 23:01:33 on dist-test-slave-6bbx"
server_key: "90cbf3f0f4cb710ff8e2165161e41467"
server_key_iv: "ab07c13a0bcc9ca0c31d646b40fc43b1"
server_key_version: "encryptionkey@0"
I20260810 23:01:33.433212 22941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:01:33.435957 22965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:33.437456 22941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:33.437592 22941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "fdf61435f1704fb08d4be117ae2b4bd0"
format_stamp: "Formatted at 2026-08-10 23:01:33 on dist-test-slave-6bbx"
server_key: "90cbf3f0f4cb710ff8e2165161e41467"
server_key_iv: "ab07c13a0bcc9ca0c31d646b40fc43b1"
server_key_version: "encryptionkey@0"
I20260810 23:01:33.437714 22941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:33.453568 22941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:33.456955 22941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:33.457167 22941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:33.457792 22941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:33.458693 22941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:33.458756 22941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:33.458815 22941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:33.458839 22941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:33.468017 22941 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:43071
I20260810 23:01:33.468057 23171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:43071 every 8 connection(s)
I20260810 23:01:33.468667 22941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:33.474000 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 22941
I20260810 23:01:33.474130 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:33.474506 15584 external_mini_cluster.cc:1428] Setting key bae1d9dadee15b25d2c83c7b4bce3e4d
Aug 10 23:01:33 dist-test-slave-6bbx krb5kdc[22498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402893, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:33.484591 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.470578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:57167 (local address 127.15.56.62:33899)
0810 23:01:33.470734 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:33.470738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:33.472077 (+ 1339us) server_negotiation.cc:407] Connection header received
0810 23:01:33.473024 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:33.473028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:33.473090 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:33.473170 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:33.475167 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.475825 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.477138 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.477455 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.477561 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.480500 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:33.480504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:33.480506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:33.480533 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:33.481983 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:33.482671 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:33.482674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:33.482675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:33.482765 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:33.483179 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:33.483181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:33.483183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:33.483381 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:33.483460 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:33.484265 (+ 805us) server_negotiation.cc:299] Negotiation successful
0810 23:01:33.484428 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:01:33.485762 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.470976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33899 (local address 127.15.56.2:57167)
0810 23:01:33.471917 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:33.471965 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:33.472802 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:33.473432 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:33.473440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:33.474193 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:33.474959 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.474982 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.475985 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.475989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.476997 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.477010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.477641 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.477645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.477813 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.478461 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:33.478479 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:33.480264 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:33.482129 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:33.482137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:33.482159 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:33.482544 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:33.482853 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:33.482856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:33.482858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:33.483067 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:33.483488 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:33.483497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:33.483923 (+ 426us) client_negotiation.cc:769] Sending connection context
0810 23:01:33.484158 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 23:01:33.484555 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":125,"threads_started":1}
I20260810 23:01:33.487015 23172 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33899
I20260810 23:01:33.487272 23172 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:33.487794 23172 heartbeater.cc:511] Master 127.15.56.62:33899 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
I20260810 23:01:33.488842 22580 ts_manager.cc:194] Registered new tserver with Master: fdf61435f1704fb08d4be117ae2b4bd0 (127.15.56.2:43071)
I20260810 23:01:33.489439 22580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:57167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:01:33.506664 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33899
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:33.628545 23182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:33.628829 23182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:33.628875 23182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:33.633028 23182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:33.633093 23182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:33.633165 23182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:33.638576 23182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:33.639685 23182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:33.640789 23182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:33.650393 23192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:33.650447 23182 server_base.cc:1034] running on GCE node
W20260810 23:01:33.650398 23191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:33.650394 23196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:33.651041 23182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:33.651711 23182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:33.652906 23182 hybrid_clock.cc:648] HybridClock initialized: now 1786402893652883 us; error 34 us; skew 500 ppm
Aug 10 23:01:33 dist-test-slave-6bbx krb5kdc[22498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402893, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:33.655841 23182 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:01:33.656596 23182 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:33.656668 23182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:33.656893 23182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:33.658929 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "fd0336c1c71c432eb359eafe06b669d4"
format_stamp: "Formatted at 2026-08-10 23:01:33 on dist-test-slave-6bbx"
server_key: "66c293c7afc3c262fa37db8432fbbcf2"
server_key_iv: "a5bf03bc011764289bc1e8c36db141e6"
server_key_version: "encryptionkey@0"
I20260810 23:01:33.659464 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "fd0336c1c71c432eb359eafe06b669d4"
format_stamp: "Formatted at 2026-08-10 23:01:33 on dist-test-slave-6bbx"
server_key: "66c293c7afc3c262fa37db8432fbbcf2"
server_key_iv: "a5bf03bc011764289bc1e8c36db141e6"
server_key_version: "encryptionkey@0"
I20260810 23:01:33.663089 23182 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:33.665522 23210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:33.666844 23182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:33.666968 23182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "fd0336c1c71c432eb359eafe06b669d4"
format_stamp: "Formatted at 2026-08-10 23:01:33 on dist-test-slave-6bbx"
server_key: "66c293c7afc3c262fa37db8432fbbcf2"
server_key_iv: "a5bf03bc011764289bc1e8c36db141e6"
server_key_version: "encryptionkey@0"
I20260810 23:01:33.667070 23182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:33.686492 23182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:33.689880 23182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:33.690078 23182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:33.690713 23182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:33.691612 23182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:33.691694 23182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:33.691763 23182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:33.691793 23182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:33.700698 23182 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:44635
I20260810 23:01:33.700722 23444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:44635 every 8 connection(s)
I20260810 23:01:33.701349 23182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:33.704561 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 23182
I20260810 23:01:33.704684 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:33.704964 15584 external_mini_cluster.cc:1428] Setting key 4ce8b9ed85e9e848d01df1ae18d196d8
Aug 10 23:01:33 dist-test-slave-6bbx krb5kdc[22498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402893, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:33.714404 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.703199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:38469 (local address 127.15.56.62:33899)
0810 23:01:33.703336 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:33.703339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:33.704596 (+ 1257us) server_negotiation.cc:407] Connection header received
0810 23:01:33.705563 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:33.705567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:33.705627 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:33.705712 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:33.707015 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.707578 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.708717 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.708931 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.708998 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.711232 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:33.711235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:33.711237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:33.711268 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:33.712599 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:33.713033 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:33.713036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:33.713037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:33.713104 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:33.713403 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:33.713407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:33.713408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:33.713599 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:33.713689 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:33.714173 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 23:01:33.714281 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:01:33.715195 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.703578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33899 (local address 127.15.56.3:38469)
0810 23:01:33.704437 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:33.704480 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:33.705335 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:33.705901 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:33.705910 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:33.706371 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:33.706885 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.706900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.707723 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.707726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.708606 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.708615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.709037 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.709040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.709190 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.709704 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:33.709720 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:33.711087 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:33.712701 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:33.712706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:33.712717 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:33.712961 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:33.713163 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:33.713165 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:33.713167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:33.713288 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:33.713700 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:33.713705 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:33.713927 (+ 222us) client_negotiation.cc:769] Sending connection context
0810 23:01:33.714079 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 23:01:33.714326 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":129,"threads_started":1}
I20260810 23:01:33.716323 23445 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33899
I20260810 23:01:33.716545 23445 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:33.716972 23445 heartbeater.cc:511] Master 127.15.56.62:33899 requested a full tablet report, sending...
I20260810 23:01:33.717974 22575 ts_manager.cc:194] Registered new tserver with Master: fd0336c1c71c432eb359eafe06b669d4 (127.15.56.3:44635)
I20260810 23:01:33.718611 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:01:33.718603 22575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:38469
I20260810 23:01:33.727488 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.720652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.15.56.62:33899)
0810 23:01:33.720764 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:33.720768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:33.720960 (+ 192us) server_negotiation.cc:407] Connection header received
0810 23:01:33.721022 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:33.721024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:33.721062 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:33.721127 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:33.721755 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.722273 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.723070 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.723296 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.723386 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.724661 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:33.724664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:33.724666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:33.724690 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:33.725932 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:33.726309 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:33.726312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:33.726313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:33.726392 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:33.726605 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:33.726607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:33.726629 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:33.726851 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:33.726962 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:33.727253 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 23:01:33.727363 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:01:33.730587 22575 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:01:33.733670 22575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:33.748060 23467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.742080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:43071 (local address 127.0.0.1:59292)
0810 23:01:33.742970 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:33.743009 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:33.743142 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:33.743921 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:33.743927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:33.743954 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:33.744092 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.744100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.745329 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.745332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.746603 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.746614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.747462 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.747466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.747568 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.747616 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 23:01:33.747762 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 23:01:33.747893 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":49,"thread_start_us":101,"threads_started":1}
I20260810 23:01:33.748221 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.742195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:44635 (local address 127.0.0.1:56984)
0810 23:01:33.743377 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:33.743391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:33.743477 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:33.744152 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:33.744155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:33.744172 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:33.744281 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.744287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.745447 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.745450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.746612 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.746623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.747890 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.747893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.747973 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.747995 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:01:33.748056 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 23:01:33.748116 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"thread_start_us":116,"threads_started":1}
I20260810 23:01:33.748066 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.742244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.15.56.2:43071)
0810 23:01:33.742916 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:33.742926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:33.743095 (+ 169us) server_negotiation.cc:407] Connection header received
0810 23:01:33.743294 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:33.743299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:33.743521 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:33.743707 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:33.744218 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.745185 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.746737 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.747346 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.747453 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.747609 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:33.747802 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:01:33.747906 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":152,"threads_started":1}
I20260810 23:01:33.748755 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.742931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.15.56.3:44635)
0810 23:01:33.743527 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:33.743539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:33.743561 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:01:33.743634 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:33.743639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:33.743870 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:33.744008 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:33.744404 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.745307 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.746766 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.747550 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.747647 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.747830 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:33.748489 (+ 659us) server_negotiation.cc:299] Negotiation successful
0810 23:01:33.748593 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":147,"threads_started":1}
I20260810 23:01:33.748993 23468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.742402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:33417 (local address 127.0.0.1:42450)
0810 23:01:33.743863 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:33.743876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:33.743974 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:33.744569 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:33.744572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:33.744590 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:33.744680 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.744685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.746043 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.746047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.747168 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.747177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.748551 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.748555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.748638 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.748661 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:01:33.748736 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 23:01:33.748868 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1405,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260810 23:01:33.749015 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.742941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.15.56.1:33417)
0810 23:01:33.743736 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:33.743742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:33.743974 (+ 232us) server_negotiation.cc:407] Connection header received
0810 23:01:33.744077 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:33.744082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:33.744281 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:33.744405 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:33.744810 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.745897 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.747320 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.748147 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.748478 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.748624 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:33.748777 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:01:33.748874 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":117,"threads_started":1}
I20260810 23:01:33.751492 23300 tablet_service.cc:1467] Processing CreateTablet for tablet 20f2db61c7e948ba88c1d26c34f913ab (DEFAULT_TABLE table=test-table [id=d2d3df035a284bbda26297323aafbef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:33.751596 23055 tablet_service.cc:1467] Processing CreateTablet for tablet 20f2db61c7e948ba88c1d26c34f913ab (DEFAULT_TABLE table=test-table [id=d2d3df035a284bbda26297323aafbef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:33.752055 22804 tablet_service.cc:1467] Processing CreateTablet for tablet 20f2db61c7e948ba88c1d26c34f913ab (DEFAULT_TABLE table=test-table [id=d2d3df035a284bbda26297323aafbef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:33.752614 23300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f2db61c7e948ba88c1d26c34f913ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:33.752925 22804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f2db61c7e948ba88c1d26c34f913ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:33.752925 23055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f2db61c7e948ba88c1d26c34f913ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:33.758622 23476 tablet_bootstrap.cc:492] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4: Bootstrap starting.
I20260810 23:01:33.760452 23479 tablet_bootstrap.cc:492] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417: Bootstrap starting.
I20260810 23:01:33.760972 23476 tablet_bootstrap.cc:654] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:33.762007 23476 log.cc:824] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:33.762355 23479 tablet_bootstrap.cc:654] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:33.762712 23480 tablet_bootstrap.cc:492] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0: Bootstrap starting.
I20260810 23:01:33.763128 23479 log.cc:824] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:33.764166 23476 tablet_bootstrap.cc:492] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4: No bootstrap required, opened a new log
I20260810 23:01:33.764380 23476 ts_tablet_manager.cc:1397] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:01:33.764576 23480 tablet_bootstrap.cc:654] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:33.765166 23479 tablet_bootstrap.cc:492] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417: No bootstrap required, opened a new log
I20260810 23:01:33.765336 23479 ts_tablet_manager.cc:1397] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:01:33.765702 23480 log.cc:824] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:33.767211 23480 tablet_bootstrap.cc:492] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0: No bootstrap required, opened a new log
I20260810 23:01:33.767379 23480 ts_tablet_manager.cc:1397] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260810 23:01:33.768642 23476 raft_consensus.cc:348] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 43071 } } peers { permanent_uuid: "fd0336c1c71c432eb359eafe06b669d4" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 44635 } } peers { permanent_uuid: "0ade91a074eb45e5b32e92d4e5d45417" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33417 } }
I20260810 23:01:33.768935 23476 raft_consensus.cc:374] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:33.768996 23476 raft_consensus.cc:729] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd0336c1c71c432eb359eafe06b669d4, State: Initialized, Role: FOLLOWER
I20260810 23:01:33.769476 23476 consensus_queue.cc:260] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 43071 } } peers { permanent_uuid: "fd0336c1c71c432eb359eafe06b669d4" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 44635 } } peers { permanent_uuid: "0ade91a074eb45e5b32e92d4e5d45417" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33417 } }
I20260810 23:01:33.769714 23479 raft_consensus.cc:348] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 43071 } } peers { permanent_uuid: "fd0336c1c71c432eb359eafe06b669d4" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 44635 } } peers { permanent_uuid: "0ade91a074eb45e5b32e92d4e5d45417" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33417 } }
I20260810 23:01:33.770002 23479 raft_consensus.cc:374] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:33.770068 23479 raft_consensus.cc:729] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ade91a074eb45e5b32e92d4e5d45417, State: Initialized, Role: FOLLOWER
I20260810 23:01:33.770390 23445 heartbeater.cc:503] Master 127.15.56.62:33899 was elected leader, sending a full tablet report...
I20260810 23:01:33.770565 23479 consensus_queue.cc:260] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 43071 } } peers { permanent_uuid: "fd0336c1c71c432eb359eafe06b669d4" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 44635 } } peers { permanent_uuid: "0ade91a074eb45e5b32e92d4e5d45417" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33417 } }
I20260810 23:01:33.770424 23480 raft_consensus.cc:348] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 43071 } } peers { permanent_uuid: "fd0336c1c71c432eb359eafe06b669d4" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 44635 } } peers { permanent_uuid: "0ade91a074eb45e5b32e92d4e5d45417" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33417 } }
I20260810 23:01:33.770766 23480 raft_consensus.cc:374] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:33.770816 23480 raft_consensus.cc:729] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf61435f1704fb08d4be117ae2b4bd0, State: Initialized, Role: FOLLOWER
I20260810 23:01:33.771209 23480 consensus_queue.cc:260] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 43071 } } peers { permanent_uuid: "fd0336c1c71c432eb359eafe06b669d4" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 44635 } } peers { permanent_uuid: "0ade91a074eb45e5b32e92d4e5d45417" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33417 } }
I20260810 23:01:33.771591 23476 ts_tablet_manager.cc:1428] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 23:01:33.771591 23479 ts_tablet_manager.cc:1428] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 23:01:33.772025 22933 heartbeater.cc:503] Master 127.15.56.62:33899 was elected leader, sending a full tablet report...
I20260810 23:01:33.772151 23480 ts_tablet_manager.cc:1428] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260810 23:01:33.772476 23172 heartbeater.cc:503] Master 127.15.56.62:33899 was elected leader, sending a full tablet report...
I20260810 23:01:33.842806 23490 raft_consensus.cc:482] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:33.843057 23490 raft_consensus.cc:504] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 43071 } } peers { permanent_uuid: "fd0336c1c71c432eb359eafe06b669d4" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 44635 } } peers { permanent_uuid: "0ade91a074eb45e5b32e92d4e5d45417" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33417 } }
I20260810 23:01:33.844276 23490 leader_election.cc:288] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdf61435f1704fb08d4be117ae2b4bd0 (127.15.56.2:43071), 0ade91a074eb45e5b32e92d4e5d45417 (127.15.56.1:33417)
I20260810 23:01:33.848613 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.844593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:33417 (local address 127.15.56.3:41567)
0810 23:01:33.844844 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:33.844860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:33.844962 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:33.845260 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:33.845263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:33.845284 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:33.845407 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.845414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.846324 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.846326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.847387 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.847397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.848172 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.848176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.848289 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.848316 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:01:33.848413 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:01:33.848493 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260810 23:01:33.848644 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.844818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:41567 (local address 127.15.56.1:33417)
0810 23:01:33.844963 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:33.844967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:33.844982 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:01:33.845046 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:33.845049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:33.845103 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:33.845183 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:33.845543 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.846199 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.847527 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.848056 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.848141 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.848202 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:33.848432 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 23:01:33.848510 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:01:33.848845 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.844593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:43071 (local address 127.15.56.3:54533)
0810 23:01:33.845330 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:33.845351 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:33.845592 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:33.845928 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:33.845930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:33.845948 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:33.846048 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.846052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.846777 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.846780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.847700 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:33.847711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.848438 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.848442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:33.848547 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.848570 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:01:33.848646 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:01:33.848716 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":315,"thread_start_us":159,"threads_started":1}
I20260810 23:01:33.848891 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.844766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:54533 (local address 127.15.56.2:43071)
0810 23:01:33.844885 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:33.844889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:33.845431 (+ 542us) server_negotiation.cc:407] Connection header received
0810 23:01:33.845746 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:33.845750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:33.845802 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:33.845881 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:33.846150 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.846664 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.847823 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.848307 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.848421 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.848480 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:33.848678 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 23:01:33.848759 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:01:33.849493 23088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f2db61c7e948ba88c1d26c34f913ab" candidate_uuid: "fd0336c1c71c432eb359eafe06b669d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" is_pre_election: true
I20260810 23:01:33.849499 22842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f2db61c7e948ba88c1d26c34f913ab" candidate_uuid: "fd0336c1c71c432eb359eafe06b669d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ade91a074eb45e5b32e92d4e5d45417" is_pre_election: true
I20260810 23:01:33.849855 23088 raft_consensus.cc:2454] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd0336c1c71c432eb359eafe06b669d4 in term 0.
I20260810 23:01:33.849845 22842 raft_consensus.cc:2454] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd0336c1c71c432eb359eafe06b669d4 in term 0.
I20260810 23:01:33.850445 23213 leader_election.cc:302] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fd0336c1c71c432eb359eafe06b669d4, fdf61435f1704fb08d4be117ae2b4bd0; no voters:
I20260810 23:01:33.850749 23490 raft_consensus.cc:2781] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:33.850841 23490 raft_consensus.cc:482] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:33.850874 23490 raft_consensus.cc:3037] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:33.852010 23490 raft_consensus.cc:504] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 43071 } } peers { permanent_uuid: "fd0336c1c71c432eb359eafe06b669d4" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 44635 } } peers { permanent_uuid: "0ade91a074eb45e5b32e92d4e5d45417" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33417 } }
I20260810 23:01:33.852368 23490 leader_election.cc:288] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [CANDIDATE]: Term 1 election: Requested vote from peers fdf61435f1704fb08d4be117ae2b4bd0 (127.15.56.2:43071), 0ade91a074eb45e5b32e92d4e5d45417 (127.15.56.1:33417)
I20260810 23:01:33.852800 22842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f2db61c7e948ba88c1d26c34f913ab" candidate_uuid: "fd0336c1c71c432eb359eafe06b669d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ade91a074eb45e5b32e92d4e5d45417"
I20260810 23:01:33.852800 23088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f2db61c7e948ba88c1d26c34f913ab" candidate_uuid: "fd0336c1c71c432eb359eafe06b669d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf61435f1704fb08d4be117ae2b4bd0"
I20260810 23:01:33.852945 22842 raft_consensus.cc:3037] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:33.852945 23088 raft_consensus.cc:3037] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:33.854279 23088 raft_consensus.cc:2454] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd0336c1c71c432eb359eafe06b669d4 in term 1.
I20260810 23:01:33.854281 22842 raft_consensus.cc:2454] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd0336c1c71c432eb359eafe06b669d4 in term 1.
I20260810 23:01:33.854640 23223 leader_election.cc:302] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ade91a074eb45e5b32e92d4e5d45417, fd0336c1c71c432eb359eafe06b669d4, fdf61435f1704fb08d4be117ae2b4bd0; no voters:
I20260810 23:01:33.854856 23490 raft_consensus.cc:2781] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:33.855104 23490 raft_consensus.cc:686] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 1 LEADER]: Becoming Leader. State: Replica: fd0336c1c71c432eb359eafe06b669d4, State: Running, Role: LEADER
I20260810 23:01:33.855466 23490 consensus_queue.cc:237] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 43071 } } peers { permanent_uuid: "fd0336c1c71c432eb359eafe06b669d4" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 44635 } } peers { permanent_uuid: "0ade91a074eb45e5b32e92d4e5d45417" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33417 } }
I20260810 23:01:33.858810 22575 catalog_manager.cc:5495] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd0336c1c71c432eb359eafe06b669d4 (127.15.56.3). New cstate: current_term: 1 leader_uuid: "fd0336c1c71c432eb359eafe06b669d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 43071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd0336c1c71c432eb359eafe06b669d4" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 44635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ade91a074eb45e5b32e92d4e5d45417" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33417 } health_report { overall_health: UNKNOWN } } }
I20260810 23:01:33.877926 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.874155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.15.56.3:44635)
0810 23:01:33.874296 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:33.874301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:33.874316 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:01:33.874492 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:33.874494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:33.874552 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:33.874667 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:33.874897 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.875568 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.876475 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:33.876693 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:33.876782 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:33.877086 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:33.877535 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:33.877647 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:33.877711 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 23:01:33.877769 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:01:33.883805 22842 raft_consensus.cc:1261] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417 [term 1 FOLLOWER]: Refusing update from remote peer fd0336c1c71c432eb359eafe06b669d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:33.883904 23088 raft_consensus.cc:1261] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0 [term 1 FOLLOWER]: Refusing update from remote peer fd0336c1c71c432eb359eafe06b669d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:33.884624 23498 consensus_queue.cc:1035] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ade91a074eb45e5b32e92d4e5d45417" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:33.884835 23490 consensus_queue.cc:1035] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf61435f1704fb08d4be117ae2b4bd0" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:33.893950 23502 mvcc.cc:205] Tried to move back new op lower bound from 7317106253340643328 to 7317106253235027968. Current Snapshot: MvccSnapshot[applied={T|T < 7317106253340643328}]
I20260810 23:01:33.902520 22575 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:33.902695 22575 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:33.905089 22575 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3c92d9297244a26b05949262c13eb03: Sending DeleteTablet for 3 replicas of tablet 20f2db61c7e948ba88c1d26c34f913ab
I20260810 23:01:33.905915 23300 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f2db61c7e948ba88c1d26c34f913ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:33 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:56984
I20260810 23:01:33.906006 23055 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f2db61c7e948ba88c1d26c34f913ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:33 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:59292
I20260810 23:01:33.906844 23520 tablet_replica.cc:330] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4: stopping tablet replica
I20260810 23:01:33.906848 22804 tablet_service.cc:1514] Processing DeleteTablet for tablet 20f2db61c7e948ba88c1d26c34f913ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:33 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:42450
I20260810 23:01:33.907390 23521 tablet_replica.cc:330] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417: stopping tablet replica
I20260810 23:01:33.907749 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 22685
I20260810 23:01:33.907779 23521 raft_consensus.cc:2229] T 20f2db61c7e948ba88c1d26c34f913ab P 0ade91a074eb45e5b32e92d4e5d45417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:33.926530 23519 tablet_replica.cc:330] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0: stopping tablet replica
I20260810 23:01:33.907227 23520 raft_consensus.cc:2229] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:33.927191 23519 raft_consensus.cc:2229] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:33.927416 23520 raft_consensus.cc:2258] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:33.927649 23519 raft_consensus.cc:2258] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:01:33.928892 22553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:33417 (error 108)
I20260810 23:01:33.929139 23520 ts_tablet_manager.cc:1905] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:33.929169 23519 ts_tablet_manager.cc:1905] T 20f2db61c7e948ba88c1d26c34f913ab P fdf61435f1704fb08d4be117ae2b4bd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:33.929736 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 22941
I20260810 23:01:33.961750 23468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.961321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:33417 (local address 127.0.0.1:42452)
0810 23:01:33.961557 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:33.961644 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:33417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260810 23:01:33.961984 22553 catalog_manager.cc:4584] TS 0ade91a074eb45e5b32e92d4e5d45417 (127.15.56.1:33417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20f2db61c7e948ba88c1d26c34f913ab: Network error: Client connection negotiation failed: client connection to 127.15.56.1:33417: connect: Connection refused (error 111)
I20260810 23:01:33.966879 23520 ts_tablet_manager.cc:1918] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:33.967121 23520 log.cc:1196] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/20f2db61c7e948ba88c1d26c34f913ab
I20260810 23:01:33.967527 23520 ts_tablet_manager.cc:1939] T 20f2db61c7e948ba88c1d26c34f913ab P fd0336c1c71c432eb359eafe06b669d4: Deleting consensus metadata
I20260810 23:01:33.969177 22552 catalog_manager.cc:4841] TS fd0336c1c71c432eb359eafe06b669d4 (127.15.56.3:44635): tablet 20f2db61c7e948ba88c1d26c34f913ab (table test-table [id=d2d3df035a284bbda26297323aafbef7]) successfully deleted
I20260810 23:01:33.972005 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 23182
I20260810 23:01:33.976972 23468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:33.972212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:43071 (local address 127.0.0.1:59298)
0810 23:01:33.976799 (+ 4587us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:33.976879 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.2:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4473}
I20260810 23:01:33.986630 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 22520
2026-08-10T23:01:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2529 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T23:01:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:34Z Disabled control of system clock
I20260810 23:01:34.023281 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:36895
--webserver_interface=127.15.56.62
--webserver_port=0
--builtin_ntp_servers=127.15.56.20:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:36895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:01:34.135874 23530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:34.136157 23530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:34.136193 23530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:34.140578 23530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:34.140640 23530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:34.140654 23530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:34.140671 23530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:34.140684 23530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:34.145468 23530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:36895
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:36895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:34.146526 23530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:34.147933 23530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:34.157069 23544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:34.157073 23542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:34.157133 23530 server_base.cc:1034] running on GCE node
W20260810 23:01:34.157069 23547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:34.157934 23530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:34.159152 23530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:34.160312 23530 hybrid_clock.cc:648] HybridClock initialized: now 1786402894160289 us; error 39 us; skew 500 ppm
I20260810 23:01:34.162196 23530 webserver.cc:460] Webserver started at http://127.15.56.62:43805/ using document root <none> and password file <none>
I20260810 23:01:34.162803 23530 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:34.162876 23530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:34.163086 23530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:34.165218 23530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "e651b60c5d4141efac1caf0e65ab2845"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "d9afed435ab5097e23435aa80a9610af"
server_key_iv: "8005478d40a24efa88cbbdf13b66bb40"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.165802 23530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "e651b60c5d4141efac1caf0e65ab2845"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "d9afed435ab5097e23435aa80a9610af"
server_key_iv: "8005478d40a24efa88cbbdf13b66bb40"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.169595 23530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:01:34.172030 23554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:34.173301 23530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:01:34.173419 23530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "e651b60c5d4141efac1caf0e65ab2845"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "d9afed435ab5097e23435aa80a9610af"
server_key_iv: "8005478d40a24efa88cbbdf13b66bb40"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.173522 23530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:34.178382 23530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:34.179005 23530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:34.179170 23530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:34.185846 23530 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:36895
I20260810 23:01:34.185879 23647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:36895 every 8 connection(s)
I20260810 23:01:34.186765 23530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:34.189284 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 23530
I20260810 23:01:34.189412 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:34.189733 15584 external_mini_cluster.cc:1428] Setting key f385c769709f23540969708220bc3a85
I20260810 23:01:34.190379 23652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:34.195186 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.191086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.15.56.62:36895)
0810 23:01:34.191739 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:34.191755 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:34.191794 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:01:34.192462 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:34.192479 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:34.192828 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:34.193137 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:34.193951 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:34.193956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:34.193968 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:34.193978 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:34.194034 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:34.194050 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:34.194179 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:34.194249 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 23:01:34.194493 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260810 23:01:34.197427 23652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845: Bootstrap starting.
I20260810 23:01:34.199867 23652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:34.200614 23652 log.cc:824] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:34.202927 23652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845: No bootstrap required, opened a new log
I20260810 23:01:34.206619 23652 raft_consensus.cc:348] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651b60c5d4141efac1caf0e65ab2845" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36895 } }
I20260810 23:01:34.206887 23652 raft_consensus.cc:374] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:34.206924 23652 raft_consensus.cc:729] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e651b60c5d4141efac1caf0e65ab2845, State: Initialized, Role: FOLLOWER
I20260810 23:01:34.207363 23652 consensus_queue.cc:260] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651b60c5d4141efac1caf0e65ab2845" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36895 } }
I20260810 23:01:34.207487 23652 raft_consensus.cc:388] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:34.207537 23652 raft_consensus.cc:482] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:34.207614 23652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:34.208681 23652 raft_consensus.cc:504] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651b60c5d4141efac1caf0e65ab2845" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36895 } }
I20260810 23:01:34.209031 23652 leader_election.cc:302] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e651b60c5d4141efac1caf0e65ab2845; no voters:
I20260810 23:01:34.209373 23652 leader_election.cc:288] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:34.209534 23662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:34.209767 23662 raft_consensus.cc:686] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [term 1 LEADER]: Becoming Leader. State: Replica: e651b60c5d4141efac1caf0e65ab2845, State: Running, Role: LEADER
I20260810 23:01:34.210107 23662 consensus_queue.cc:237] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651b60c5d4141efac1caf0e65ab2845" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36895 } }
I20260810 23:01:34.210845 23652 sys_catalog.cc:564] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:34.211946 23666 sys_catalog.cc:455] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e651b60c5d4141efac1caf0e65ab2845. Latest consensus state: current_term: 1 leader_uuid: "e651b60c5d4141efac1caf0e65ab2845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651b60c5d4141efac1caf0e65ab2845" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36895 } } }
I20260810 23:01:34.211961 23663 sys_catalog.cc:455] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e651b60c5d4141efac1caf0e65ab2845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e651b60c5d4141efac1caf0e65ab2845" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36895 } } }
I20260810 23:01:34.212208 23663 sys_catalog.cc:458] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:34.212244 23666 sys_catalog.cc:458] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:34.212602 23680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:34.215910 23680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:34.221164 23680 catalog_manager.cc:1347] Generated new cluster ID: 5ceeb7b7179b490e99da1e348a5346b1
I20260810 23:01:34.221235 23680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:34.227921 23680 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:34.228246 23680 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:34.237903 23680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845: Generated new TSK 0
I20260810 23:01:34.238610 23680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:01:34.243587 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:36895
--builtin_ntp_servers=127.15.56.20:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:01:34.377200 23695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:34.377478 23695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:34.377518 23695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:34.381763 23695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:34.381834 23695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:34.381891 23695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:34.386996 23695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:36895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:34.388121 23695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:34.389523 23695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:34.399482 23707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:34.399557 23695 server_base.cc:1034] running on GCE node
W20260810 23:01:34.399829 23706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:34.399833 23709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:34.400324 23695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:34.401130 23695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:34.402292 23695 hybrid_clock.cc:648] HybridClock initialized: now 1786402894402261 us; error 43 us; skew 500 ppm
I20260810 23:01:34.404223 23695 webserver.cc:460] Webserver started at http://127.15.56.1:45591/ using document root <none> and password file <none>
I20260810 23:01:34.404811 23695 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:34.404877 23695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:34.405092 23695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:34.407208 23695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "d2f749cba3ba4f7d96c36837863a0e82"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "00367056d9a41731ab6a7431116b1c3f"
server_key_iv: "d4299f385f6863408eba290191a65197"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.407804 23695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "d2f749cba3ba4f7d96c36837863a0e82"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "00367056d9a41731ab6a7431116b1c3f"
server_key_iv: "d4299f385f6863408eba290191a65197"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.411623 23695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:01:34.414376 23722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:34.415628 23695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:34.415783 23695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "d2f749cba3ba4f7d96c36837863a0e82"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "00367056d9a41731ab6a7431116b1c3f"
server_key_iv: "d4299f385f6863408eba290191a65197"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.415889 23695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:34.420445 23695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:34.421074 23695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:34.421236 23695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:34.421764 23695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:34.422688 23695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:34.422736 23695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:34.422794 23695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:34.422821 23695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:34.431437 23695 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:37449
I20260810 23:01:34.431473 23939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:37449 every 8 connection(s)
I20260810 23:01:34.432389 23695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:34.438464 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.434402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:56505 (local address 127.15.56.62:36895)
0810 23:01:34.434564 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:34.434569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:34.435644 (+ 1075us) server_negotiation.cc:407] Connection header received
0810 23:01:34.436999 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:34.437002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:34.437040 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:34.437101 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:34.438132 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:34.438135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:34.438137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:34.438145 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:34.438160 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:34.438166 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:34.438229 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:34.438317 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 23:01:34.438354 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:01:34.439339 23944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.434841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:36895 (local address 127.15.56.1:56505)
0810 23:01:34.435498 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:34.435538 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:34.436794 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:34.437246 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:34.437252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:34.437349 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:34.437898 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:01:34.437899 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:34.437943 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:01:34.438014 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:34.438219 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:34.438224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:34.438227 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:01:34.438347 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 23:01:34.438548 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260810 23:01:34.440519 23942 heartbeater.cc:347] Connected to a master server at 127.15.56.62:36895
I20260810 23:01:34.440771 23942 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:34.441174 23942 heartbeater.cc:511] Master 127.15.56.62:36895 requested a full tablet report, sending...
I20260810 23:01:34.441825 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 23695
I20260810 23:01:34.441941 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:34.442231 15584 external_mini_cluster.cc:1428] Setting key 2a1c5a7cf38e3d1b81405e1b3b413615
I20260810 23:01:34.442739 23581 ts_manager.cc:194] Registered new tserver with Master: d2f749cba3ba4f7d96c36837863a0e82 (127.15.56.1:37449)
I20260810 23:01:34.444494 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:36895
--builtin_ntp_servers=127.15.56.20:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:01:34.560640 23947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:34.560925 23947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:34.560983 23947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:34.565102 23947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:34.565164 23947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:34.565246 23947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:34.570050 23947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:36895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:34.571182 23947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:34.572698 23947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:34.582780 23959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:34.582780 23963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:34.582783 23957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:34.582824 23947 server_base.cc:1034] running on GCE node
I20260810 23:01:34.583459 23947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:34.584187 23947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:34.585366 23947 hybrid_clock.cc:648] HybridClock initialized: now 1786402894585330 us; error 46 us; skew 500 ppm
I20260810 23:01:34.587364 23947 webserver.cc:460] Webserver started at http://127.15.56.2:43989/ using document root <none> and password file <none>
I20260810 23:01:34.588033 23947 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:34.588111 23947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:34.588344 23947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:34.590515 23947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "866208c36bb2424f819bea2913f317be"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "04e0b9ec69dee2d44421ab9faa1abadf"
server_key_iv: "904193a4dbd3d8b869282081546117d6"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.591104 23947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "866208c36bb2424f819bea2913f317be"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "04e0b9ec69dee2d44421ab9faa1abadf"
server_key_iv: "904193a4dbd3d8b869282081546117d6"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.594973 23947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:01:34.597630 23980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:34.598978 23947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:34.599114 23947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "866208c36bb2424f819bea2913f317be"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "04e0b9ec69dee2d44421ab9faa1abadf"
server_key_iv: "904193a4dbd3d8b869282081546117d6"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.599248 23947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:34.604167 23947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:34.604830 23947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:34.605002 23947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:34.605594 23947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:34.606607 23947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:34.606657 23947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:34.606714 23947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:34.606739 23947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:34.616039 23947 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:38697
I20260810 23:01:34.616072 24171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:38697 every 8 connection(s)
I20260810 23:01:34.617010 23947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:34.621775 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 23947
I20260810 23:01:34.621923 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:34.622219 15584 external_mini_cluster.cc:1428] Setting key 2eca93c643f4c8fe6e0b81b5803090f5
I20260810 23:01:34.623608 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.619904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:52537 (local address 127.15.56.62:36895)
0810 23:01:34.620045 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:34.620050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:34.620330 (+ 280us) server_negotiation.cc:407] Connection header received
0810 23:01:34.621656 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:34.621660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:34.621709 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:34.621810 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:34.623080 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:34.623083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:34.623085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:34.623096 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:34.623130 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:34.623138 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:34.623217 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:34.623415 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 23:01:34.623458 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:01:34.624485 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:36895
--builtin_ntp_servers=127.15.56.20:41929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 23:01:34.624799 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.619344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:36895 (local address 127.15.56.2:52537)
0810 23:01:34.620191 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:34.620235 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:34.621427 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:34.622051 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:34.622060 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:34.622180 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:34.622808 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:01:34.622810 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:34.622855 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:01:34.622931 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:34.623227 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:34.623237 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:34.623243 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:01:34.623423 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 23:01:34.623713 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":117,"threads_started":1}
I20260810 23:01:34.625948 24173 heartbeater.cc:347] Connected to a master server at 127.15.56.62:36895
I20260810 23:01:34.626204 24173 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:34.626636 24173 heartbeater.cc:511] Master 127.15.56.62:36895 requested a full tablet report, sending...
I20260810 23:01:34.627748 23581 ts_manager.cc:194] Registered new tserver with Master: 866208c36bb2424f819bea2913f317be (127.15.56.2:38697)
W20260810 23:01:34.745220 24181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:34.745540 24181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:34.745581 24181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:34.749836 24181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:34.749912 24181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:34.749972 24181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:34.755097 24181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:41929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:36895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:34.756362 24181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:34.757822 24181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:01:34.767625 24181 server_base.cc:1034] running on GCE node
W20260810 23:01:34.767719 24190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:34.767719 24192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:34.767807 24195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:34.768319 24181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:34.768963 24181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:34.770170 24181 hybrid_clock.cc:648] HybridClock initialized: now 1786402894770141 us; error 40 us; skew 500 ppm
I20260810 23:01:34.772076 24181 webserver.cc:460] Webserver started at http://127.15.56.3:35677/ using document root <none> and password file <none>
I20260810 23:01:34.772670 24181 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:34.772742 24181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:34.772956 24181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:34.775084 24181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "22c5d9f82c6048c58f0d330793a5dd8c"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "bf146feda18078add9c7c359fd928b07"
server_key_iv: "df99e42cecd4448e4f0ee3a0843884e0"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.775684 24181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "22c5d9f82c6048c58f0d330793a5dd8c"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "bf146feda18078add9c7c359fd928b07"
server_key_iv: "df99e42cecd4448e4f0ee3a0843884e0"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.779841 24181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:01:34.782423 24206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:34.783818 24181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:01:34.783931 24181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "22c5d9f82c6048c58f0d330793a5dd8c"
format_stamp: "Formatted at 2026-08-10 23:01:34 on dist-test-slave-6bbx"
server_key: "bf146feda18078add9c7c359fd928b07"
server_key_iv: "df99e42cecd4448e4f0ee3a0843884e0"
server_key_version: "encryptionkey@0"
I20260810 23:01:34.784051 24181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:34.788784 24181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:34.789426 24181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:34.789592 24181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:34.790141 24181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:34.791131 24181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:34.791182 24181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:34.791235 24181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:34.791261 24181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:34.800040 24181 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:40723
I20260810 23:01:34.800065 24408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:40723 every 8 connection(s)
I20260810 23:01:34.800999 24181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:34.802332 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 24181
I20260810 23:01:34.802443 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:34.802755 15584 external_mini_cluster.cc:1428] Setting key 953e45c78baa5287f3ede973d7b8a12d
I20260810 23:01:34.807437 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.802994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:51773 (local address 127.15.56.62:36895)
0810 23:01:34.803132 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:34.803137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:34.804419 (+ 1282us) server_negotiation.cc:407] Connection header received
0810 23:01:34.805819 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:34.805822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:34.805867 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:34.805939 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:34.806997 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:34.807001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:34.807004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:34.807017 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:34.807039 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:34.807047 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:34.807119 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:34.807255 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:01:34.807306 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:01:34.808338 24412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.803455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:36895 (local address 127.15.56.3:51773)
0810 23:01:34.804283 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:34.804327 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:34.805616 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:34.806106 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:34.806113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:34.806202 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:34.806734 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:01:34.806736 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:34.806786 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:01:34.806856 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:34.807117 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:34.807126 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:34.807132 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:01:34.807293 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:01:34.807540 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":158,"threads_started":1}
I20260810 23:01:34.809446 24409 heartbeater.cc:347] Connected to a master server at 127.15.56.62:36895
I20260810 23:01:34.809652 24409 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:34.810029 24409 heartbeater.cc:511] Master 127.15.56.62:36895 requested a full tablet report, sending...
I20260810 23:01:34.810968 23581 ts_manager.cc:194] Registered new tserver with Master: 22c5d9f82c6048c58f0d330793a5dd8c (127.15.56.3:40723)
I20260810 23:01:34.816609 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:01:34.820053 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.818406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.15.56.62:36895)
0810 23:01:34.818524 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:34.818529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:34.818706 (+ 177us) server_negotiation.cc:407] Connection header received
0810 23:01:34.818840 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:34.818843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:34.818876 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:34.818938 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:34.819748 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:34.819751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:34.819753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:34.819764 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:34.819784 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:34.819792 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:34.819858 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:34.819920 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 23:01:34.819953 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:01:34.822739 23581 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:01:34.825743 23581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:34.837498 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.834412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:40723 (local address 127.0.0.1:45426)
0810 23:01:34.835199 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:34.835211 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:34.835272 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:34.836076 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:34.836079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:34.836122 (+ 43us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:34.836727 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:01:34.836729 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:34.836750 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:01:34.836831 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:34.837133 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:34.837143 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:34.837162 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 23:01:34.837273 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 23:01:34.837349 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":88,"threads_started":1}
I20260810 23:01:34.837512 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.835157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.15.56.3:40723)
0810 23:01:34.835632 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:34.835655 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:34.835704 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:01:34.835759 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:34.835765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:34.835937 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:34.836446 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:34.836970 (+ 524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:34.836976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:34.836989 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:34.836999 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:34.837034 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:34.837053 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:34.837147 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:34.837298 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 23:01:34.837379 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":150,"threads_started":1}
I20260810 23:01:34.837718 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.834088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:37449 (local address 127.0.0.1:51408)
0810 23:01:34.834920 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:34.834970 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:34.835074 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:34.836273 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:34.836276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:34.836288 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:34.837034 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:01:34.837036 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:34.837047 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:01:34.837071 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:34.837475 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:34.837478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:34.837481 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:01:34.837550 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 23:01:34.837587 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":130,"threads_started":1}
I20260810 23:01:34.838193 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.834260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.15.56.1:37449)
0810 23:01:34.834986 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:34.835001 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:34.835029 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:01:34.835213 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:34.835220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:34.835432 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:34.835564 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:34.837232 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:34.837237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:34.837247 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:34.837259 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:34.837296 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:34.837326 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:34.837432 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:34.837956 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 23:01:34.838034 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":103,"threads_started":1}
I20260810 23:01:34.838516 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.834106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:38697 (local address 127.0.0.1:53918)
0810 23:01:34.835003 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:34.835017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:34.835102 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:34.836037 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:34.836044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:34.836123 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:34.836840 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:01:34.836842 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:34.836854 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:01:34.836889 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:34.837475 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:34.837478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:34.837481 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:01:34.837550 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 23:01:34.838415 (+ 865us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":90,"threads_started":1}
I20260810 23:01:34.838740 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:34.834776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.15.56.2:38697)
0810 23:01:34.835520 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:34.835532 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:34.835554 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:01:34.835603 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:34.835608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:34.835876 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:34.836036 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:34.837230 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:34.837236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:34.837247 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:34.837258 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:34.837296 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:34.837322 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:34.837432 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:34.838104 (+ 672us) server_negotiation.cc:299] Negotiation successful
0810 23:01:34.838164 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":115,"threads_started":1}
I20260810 23:01:34.840924 23822 tablet_service.cc:1467] Processing CreateTablet for tablet cac2312a62af47b6a2deb0d8194b5b1a (DEFAULT_TABLE table=test-table [id=7685383a9d9241d1b6d0641824f104ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:34.841106 24066 tablet_service.cc:1467] Processing CreateTablet for tablet cac2312a62af47b6a2deb0d8194b5b1a (DEFAULT_TABLE table=test-table [id=7685383a9d9241d1b6d0641824f104ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:34.841091 24292 tablet_service.cc:1467] Processing CreateTablet for tablet cac2312a62af47b6a2deb0d8194b5b1a (DEFAULT_TABLE table=test-table [id=7685383a9d9241d1b6d0641824f104ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:34.842047 23822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac2312a62af47b6a2deb0d8194b5b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:34.842118 24292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac2312a62af47b6a2deb0d8194b5b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:34.842118 24066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac2312a62af47b6a2deb0d8194b5b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:34.847923 24443 tablet_bootstrap.cc:492] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82: Bootstrap starting.
I20260810 23:01:34.849202 24446 tablet_bootstrap.cc:492] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: Bootstrap starting.
I20260810 23:01:34.849962 24445 tablet_bootstrap.cc:492] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: Bootstrap starting.
I20260810 23:01:34.850665 24443 tablet_bootstrap.cc:654] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:34.851315 24446 tablet_bootstrap.cc:654] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:34.851564 24443 log.cc:824] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:34.851702 24445 tablet_bootstrap.cc:654] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:34.852133 24446 log.cc:824] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:34.852489 24445 log.cc:824] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:34.853695 24443 tablet_bootstrap.cc:492] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82: No bootstrap required, opened a new log
I20260810 23:01:34.853899 24443 ts_tablet_manager.cc:1397] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260810 23:01:34.854022 24446 tablet_bootstrap.cc:492] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: No bootstrap required, opened a new log
I20260810 23:01:34.854135 24445 tablet_bootstrap.cc:492] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: No bootstrap required, opened a new log
I20260810 23:01:34.854203 24446 ts_tablet_manager.cc:1397] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:01:34.854318 24445 ts_tablet_manager.cc:1397] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:01:34.857805 24445 raft_consensus.cc:348] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f749cba3ba4f7d96c36837863a0e82" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37449 } } peers { permanent_uuid: "866208c36bb2424f819bea2913f317be" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 38697 } } peers { permanent_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40723 } }
I20260810 23:01:34.858026 24445 raft_consensus.cc:374] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:34.858088 24445 raft_consensus.cc:729] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 866208c36bb2424f819bea2913f317be, State: Initialized, Role: FOLLOWER
I20260810 23:01:34.858345 24443 raft_consensus.cc:348] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f749cba3ba4f7d96c36837863a0e82" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37449 } } peers { permanent_uuid: "866208c36bb2424f819bea2913f317be" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 38697 } } peers { permanent_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40723 } }
I20260810 23:01:34.858345 24446 raft_consensus.cc:348] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f749cba3ba4f7d96c36837863a0e82" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37449 } } peers { permanent_uuid: "866208c36bb2424f819bea2913f317be" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 38697 } } peers { permanent_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40723 } }
I20260810 23:01:34.858506 24445 consensus_queue.cc:260] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f749cba3ba4f7d96c36837863a0e82" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37449 } } peers { permanent_uuid: "866208c36bb2424f819bea2913f317be" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 38697 } } peers { permanent_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40723 } }
I20260810 23:01:34.858596 24443 raft_consensus.cc:374] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:34.858596 24446 raft_consensus.cc:374] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:34.858664 24446 raft_consensus.cc:729] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c5d9f82c6048c58f0d330793a5dd8c, State: Initialized, Role: FOLLOWER
I20260810 23:01:34.858664 24443 raft_consensus.cc:729] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f749cba3ba4f7d96c36837863a0e82, State: Initialized, Role: FOLLOWER
I20260810 23:01:34.859153 24173 heartbeater.cc:503] Master 127.15.56.62:36895 was elected leader, sending a full tablet report...
I20260810 23:01:34.859108 24446 consensus_queue.cc:260] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f749cba3ba4f7d96c36837863a0e82" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37449 } } peers { permanent_uuid: "866208c36bb2424f819bea2913f317be" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 38697 } } peers { permanent_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40723 } }
I20260810 23:01:34.859511 24445 ts_tablet_manager.cc:1428] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 23:01:34.859817 24409 heartbeater.cc:503] Master 127.15.56.62:36895 was elected leader, sending a full tablet report...
I20260810 23:01:34.860086 24446 ts_tablet_manager.cc:1428] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:01:34.860320 24443 consensus_queue.cc:260] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f749cba3ba4f7d96c36837863a0e82" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37449 } } peers { permanent_uuid: "866208c36bb2424f819bea2913f317be" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 38697 } } peers { permanent_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40723 } }
I20260810 23:01:34.860919 23942 heartbeater.cc:503] Master 127.15.56.62:36895 was elected leader, sending a full tablet report...
I20260810 23:01:34.861222 24443 ts_tablet_manager.cc:1428] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
W20260810 23:01:34.868003 24178 tablet.cc:2369] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:01:34.933512 23943 tablet.cc:2369] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:01:35.051987 24411 tablet.cc:2369] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:01:35.237267 24453 raft_consensus.cc:482] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:35.237519 24453 raft_consensus.cc:504] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f749cba3ba4f7d96c36837863a0e82" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37449 } } peers { permanent_uuid: "866208c36bb2424f819bea2913f317be" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 38697 } } peers { permanent_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40723 } }
I20260810 23:01:35.238741 24453 leader_election.cc:288] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 866208c36bb2424f819bea2913f317be (127.15.56.2:38697), 22c5d9f82c6048c58f0d330793a5dd8c (127.15.56.3:40723)
I20260810 23:01:35.240998 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:35.239166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:42549 (local address 127.15.56.3:40723)
0810 23:01:35.239334 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:35.239339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:35.239570 (+ 231us) server_negotiation.cc:407] Connection header received
0810 23:01:35.239704 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:35.239708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:35.239747 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:35.239821 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:35.240574 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:35.240577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:35.240580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:35.240593 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:35.240617 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:35.240627 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:35.240725 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:35.240805 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 23:01:35.240852 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:01:35.241021 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:35.239081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:40723 (local address 127.15.56.1:42549)
0810 23:01:35.239476 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:35.239488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:35.239593 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:35.239847 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:35.239850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:35.239865 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:35.240369 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:01:35.240372 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:35.240385 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:01:35.240419 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:35.240718 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:35.240721 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:35.240724 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:01:35.240812 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:01:35.240852 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":111,"threads_started":1}
I20260810 23:01:35.241074 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:35.239241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:57907 (local address 127.15.56.2:38697)
0810 23:01:35.239391 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:35.239395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:35.239621 (+ 226us) server_negotiation.cc:407] Connection header received
0810 23:01:35.239763 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:35.239766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:35.239802 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:35.239870 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:35.240701 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:35.240705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:35.240708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:35.240722 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:35.240745 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:35.240756 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:35.240839 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:35.240930 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 23:01:35.240966 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:01:35.241096 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:35.239085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:38697 (local address 127.15.56.1:57907)
0810 23:01:35.239525 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:35.239539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:35.239631 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:35.239908 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:35.239911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:35.239925 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:35.240515 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:01:35.240518 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:35.240529 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:01:35.240560 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:35.240848 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:35.240852 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:35.240855 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:01:35.240933 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:01:35.240965 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":151,"thread_start_us":81,"threads_started":1}
I20260810 23:01:35.241845 24092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac2312a62af47b6a2deb0d8194b5b1a" candidate_uuid: "d2f749cba3ba4f7d96c36837863a0e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "866208c36bb2424f819bea2913f317be" is_pre_election: true
I20260810 23:01:35.241861 24330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac2312a62af47b6a2deb0d8194b5b1a" candidate_uuid: "d2f749cba3ba4f7d96c36837863a0e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" is_pre_election: true
I20260810 23:01:35.242224 24092 raft_consensus.cc:2454] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2f749cba3ba4f7d96c36837863a0e82 in term 0.
I20260810 23:01:35.242224 24330 raft_consensus.cc:2454] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2f749cba3ba4f7d96c36837863a0e82 in term 0.
I20260810 23:01:35.242763 23723 leader_election.cc:302] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 866208c36bb2424f819bea2913f317be, d2f749cba3ba4f7d96c36837863a0e82; no voters:
I20260810 23:01:35.243081 24453 raft_consensus.cc:2781] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:35.243175 24453 raft_consensus.cc:482] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:35.243216 24453 raft_consensus.cc:3037] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:35.244365 24453 raft_consensus.cc:504] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f749cba3ba4f7d96c36837863a0e82" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37449 } } peers { permanent_uuid: "866208c36bb2424f819bea2913f317be" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 38697 } } peers { permanent_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40723 } }
I20260810 23:01:35.244724 24453 leader_election.cc:288] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [CANDIDATE]: Term 1 election: Requested vote from peers 866208c36bb2424f819bea2913f317be (127.15.56.2:38697), 22c5d9f82c6048c58f0d330793a5dd8c (127.15.56.3:40723)
I20260810 23:01:35.245167 24092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac2312a62af47b6a2deb0d8194b5b1a" candidate_uuid: "d2f749cba3ba4f7d96c36837863a0e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "866208c36bb2424f819bea2913f317be"
I20260810 23:01:35.245167 24330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac2312a62af47b6a2deb0d8194b5b1a" candidate_uuid: "d2f749cba3ba4f7d96c36837863a0e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c5d9f82c6048c58f0d330793a5dd8c"
I20260810 23:01:35.245304 24092 raft_consensus.cc:3037] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:35.245297 24330 raft_consensus.cc:3037] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:35.246402 24330 raft_consensus.cc:2454] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2f749cba3ba4f7d96c36837863a0e82 in term 1.
I20260810 23:01:35.246404 24092 raft_consensus.cc:2454] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2f749cba3ba4f7d96c36837863a0e82 in term 1.
I20260810 23:01:35.246702 23723 leader_election.cc:302] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 866208c36bb2424f819bea2913f317be, d2f749cba3ba4f7d96c36837863a0e82; no voters:
I20260810 23:01:35.246881 24453 raft_consensus.cc:2781] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:35.247097 24453 raft_consensus.cc:686] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [term 1 LEADER]: Becoming Leader. State: Replica: d2f749cba3ba4f7d96c36837863a0e82, State: Running, Role: LEADER
I20260810 23:01:35.247542 24453 consensus_queue.cc:237] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f749cba3ba4f7d96c36837863a0e82" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37449 } } peers { permanent_uuid: "866208c36bb2424f819bea2913f317be" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 38697 } } peers { permanent_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40723 } }
I20260810 23:01:35.250917 23581 catalog_manager.cc:5495] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2f749cba3ba4f7d96c36837863a0e82 (127.15.56.1). New cstate: current_term: 1 leader_uuid: "d2f749cba3ba4f7d96c36837863a0e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f749cba3ba4f7d96c36837863a0e82" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "866208c36bb2424f819bea2913f317be" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40723 } health_report { overall_health: UNKNOWN } } }
I20260810 23:01:35.289494 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:35.287830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.15.56.1:37449)
0810 23:01:35.287956 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:35.287960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:35.287974 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:01:35.288138 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:35.288141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:35.288210 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:35.288283 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:35.289067 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:35.289070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:35.289072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:35.289084 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:35.289105 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:35.289113 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:35.289211 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:35.289337 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 23:01:35.289375 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:01:35.296335 24330 raft_consensus.cc:1261] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c [term 1 FOLLOWER]: Refusing update from remote peer d2f749cba3ba4f7d96c36837863a0e82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:35.296335 24092 raft_consensus.cc:1261] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be [term 1 FOLLOWER]: Refusing update from remote peer d2f749cba3ba4f7d96c36837863a0e82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:35.297137 24457 consensus_queue.cc:1035] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "866208c36bb2424f819bea2913f317be" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:35.297331 24453 consensus_queue.cc:1035] T cac2312a62af47b6a2deb0d8194b5b1a P d2f749cba3ba4f7d96c36837863a0e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c5d9f82c6048c58f0d330793a5dd8c" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:35.305536 24463 mvcc.cc:205] Tried to move back new op lower bound from 7317106259124781056 to 7317106258936909824. Current Snapshot: MvccSnapshot[applied={T|T < 7317106259124781056}]
I20260810 23:01:35.314177 23578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:35.314355 23578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:35.316910 23578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e651b60c5d4141efac1caf0e65ab2845: Sending DeleteTablet for 3 replicas of tablet cac2312a62af47b6a2deb0d8194b5b1a
I20260810 23:01:35.317857 24292 tablet_service.cc:1514] Processing DeleteTablet for tablet cac2312a62af47b6a2deb0d8194b5b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:35 UTC) from {username='slave'} at 127.0.0.1:45426
I20260810 23:01:35.317859 23822 tablet_service.cc:1514] Processing DeleteTablet for tablet cac2312a62af47b6a2deb0d8194b5b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:35 UTC) from {username='slave'} at 127.0.0.1:51408
I20260810 23:01:35.318397 24473 tablet_replica.cc:330] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: stopping tablet replica
I20260810 23:01:35.318440 24066 tablet_service.cc:1514] Processing DeleteTablet for tablet cac2312a62af47b6a2deb0d8194b5b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:35 UTC) from {username='slave'} at 127.0.0.1:53918
I20260810 23:01:35.318603 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 23695
I20260810 23:01:35.318945 24473 raft_consensus.cc:2229] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:35.319473 24473 raft_consensus.cc:2258] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:35.324185 24477 tablet_replica.cc:330] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: stopping tablet replica
I20260810 23:01:35.324599 24473 ts_tablet_manager.cc:1905] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:35.324645 24477 raft_consensus.cc:2229] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:35.325091 24477 raft_consensus.cc:2258] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:35.328246 24473 ts_tablet_manager.cc:1918] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:35.328447 24473 log.cc:1196] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/cac2312a62af47b6a2deb0d8194b5b1a
I20260810 23:01:35.328840 24473 ts_tablet_manager.cc:1939] T cac2312a62af47b6a2deb0d8194b5b1a P 22c5d9f82c6048c58f0d330793a5dd8c: Deleting consensus metadata
I20260810 23:01:35.329047 24477 ts_tablet_manager.cc:1905] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:35.330374 23558 catalog_manager.cc:4841] TS 22c5d9f82c6048c58f0d330793a5dd8c (127.15.56.3:40723): tablet cac2312a62af47b6a2deb0d8194b5b1a (table test-table [id=7685383a9d9241d1b6d0641824f104ed]) successfully deleted
I20260810 23:01:35.331545 24477 ts_tablet_manager.cc:1918] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:35.331655 24477 log.cc:1196] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/cac2312a62af47b6a2deb0d8194b5b1a
I20260810 23:01:35.331951 24477 ts_tablet_manager.cc:1939] T cac2312a62af47b6a2deb0d8194b5b1a P 866208c36bb2424f819bea2913f317be: Deleting consensus metadata
I20260810 23:01:35.333092 23556 catalog_manager.cc:4841] TS 866208c36bb2424f819bea2913f317be (127.15.56.2:38697): tablet cac2312a62af47b6a2deb0d8194b5b1a (table test-table [id=7685383a9d9241d1b6d0641824f104ed]) successfully deleted
W20260810 23:01:35.335000 23559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:37449 (error 108)
I20260810 23:01:35.335112 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 23947
I20260810 23:01:35.344077 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:35.343466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:37449 (local address 127.0.0.1:51422)
0810 23:01:35.343847 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:35.343955 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:37449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":85,"threads_started":1}
W20260810 23:01:35.344336 23559 catalog_manager.cc:4584] TS d2f749cba3ba4f7d96c36837863a0e82 (127.15.56.1:37449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cac2312a62af47b6a2deb0d8194b5b1a: Network error: Client connection negotiation failed: client connection to 127.15.56.1:37449: connect: Connection refused (error 111)
I20260810 23:01:35.351218 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 24181
I20260810 23:01:35.370023 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 23530
2026-08-10T23:01:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1380 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 23:01:35.395778 15584 mini_oidc.cc:152] Starting JWKS server
I20260810 23:01:35.398278 15584 webserver.cc:460] Webserver started at https://127.0.0.1:36297/ using document root <none> and password file <none>
I20260810 23:01:35.398432 15584 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:01:35.398669 15584 webserver.cc:460] Webserver started at http://127.0.0.1:33509/ using document root <none> and password file <none>
2026-08-10T23:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:35Z Disabled control of system clock
I20260810 23:01:35.409947 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:36857
--webserver_interface=127.15.56.62
--webserver_port=0
--builtin_ntp_servers=127.15.56.20:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:36857
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36297/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/testchainca.cert with env {}
W20260810 23:01:35.530153 24490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:35.530467 24490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:35.530509 24490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:35.534816 24490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:35.534897 24490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:35.534911 24490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:35.534927 24490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:35.534941 24490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:01:35.534955 24490 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:01:35.534967 24490 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36297/jwks.json
I20260810 23:01:35.540639 24490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:36857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:36857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36297/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:35.541782 24490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:35.542834 24490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:35.551789 24505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:35.551951 24490 server_base.cc:1034] running on GCE node
W20260810 23:01:35.551790 24502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:35.551786 24501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:35.552829 24490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:35.554116 24490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:35.555297 24490 hybrid_clock.cc:648] HybridClock initialized: now 1786402895555278 us; error 29 us; skew 500 ppm
I20260810 23:01:35.561092 24490 webserver.cc:460] Webserver started at http://127.15.56.62:35999/ using document root <none> and password file <none>
I20260810 23:01:35.561729 24490 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:35.561798 24490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:35.562047 24490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:35.564373 24490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "89edf14b10a644358ebd7ce29736a522"
format_stamp: "Formatted at 2026-08-10 23:01:35 on dist-test-slave-6bbx"
server_key: "a965dd5236660748f61d2eb59f558417"
server_key_iv: "7c2af87782956a92eaac1832d4b2da4b"
server_key_version: "encryptionkey@0"
I20260810 23:01:35.564990 24490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "89edf14b10a644358ebd7ce29736a522"
format_stamp: "Formatted at 2026-08-10 23:01:35 on dist-test-slave-6bbx"
server_key: "a965dd5236660748f61d2eb59f558417"
server_key_iv: "7c2af87782956a92eaac1832d4b2da4b"
server_key_version: "encryptionkey@0"
I20260810 23:01:35.569079 24490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:01:35.571774 24513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:35.573194 24490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:01:35.573359 24490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "89edf14b10a644358ebd7ce29736a522"
format_stamp: "Formatted at 2026-08-10 23:01:35 on dist-test-slave-6bbx"
server_key: "a965dd5236660748f61d2eb59f558417"
server_key_iv: "7c2af87782956a92eaac1832d4b2da4b"
server_key_version: "encryptionkey@0"
I20260810 23:01:35.573493 24490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:35.588629 24490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:35.589433 24490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:35.589627 24490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:35.596686 24490 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:36857
I20260810 23:01:35.596712 24608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:36857 every 8 connection(s)
I20260810 23:01:35.597681 24490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:35.601200 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:35.607227 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 24490
I20260810 23:01:35.607365 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:35.607373 24609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522: Bootstrap starting.
I20260810 23:01:35.607626 15584 external_mini_cluster.cc:1428] Setting key 834ff7781c4c2d62dc37049fb57fae3d
I20260810 23:01:35.610080 24609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:35.610879 24609 log.cc:824] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:35.613013 24609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522: No bootstrap required, opened a new log
I20260810 23:01:35.617316 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:35.609075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.15.56.62:36857)
0810 23:01:35.609760 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:35.609783 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:35.609820 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:01:35.610511 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:35.610531 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:35.610966 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:35.611309 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:35.611809 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:35.612790 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:35.613979 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:35.614464 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:35.614583 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:35.615474 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:35.615479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:35.615493 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:35.615945 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:35.616005 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:35.616027 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:35.616103 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:35.616257 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 23:01:35.616526 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":145,"threads_started":1}
I20260810 23:01:35.617291 24609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89edf14b10a644358ebd7ce29736a522" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36857 } }
I20260810 23:01:35.617543 24609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:35.617590 24609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89edf14b10a644358ebd7ce29736a522, State: Initialized, Role: FOLLOWER
I20260810 23:01:35.618005 24609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89edf14b10a644358ebd7ce29736a522" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36857 } }
I20260810 23:01:35.618119 24609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:35.618166 24609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:35.618232 24609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:35.619148 24609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89edf14b10a644358ebd7ce29736a522" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36857 } }
I20260810 23:01:35.619529 24609 leader_election.cc:302] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89edf14b10a644358ebd7ce29736a522; no voters:
I20260810 23:01:35.619899 24609 leader_election.cc:288] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:35.619982 24619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:35.620186 24619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [term 1 LEADER]: Becoming Leader. State: Replica: 89edf14b10a644358ebd7ce29736a522, State: Running, Role: LEADER
I20260810 23:01:35.620231 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:01:35.620502 24619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89edf14b10a644358ebd7ce29736a522" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36857 } }
I20260810 23:01:35.621333 24609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:35.621646 24619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89edf14b10a644358ebd7ce29736a522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89edf14b10a644358ebd7ce29736a522" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36857 } } }
I20260810 23:01:35.621766 24619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:35.621973 24619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89edf14b10a644358ebd7ce29736a522. Latest consensus state: current_term: 1 leader_uuid: "89edf14b10a644358ebd7ce29736a522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89edf14b10a644358ebd7ce29736a522" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 36857 } } }
I20260810 23:01:35.622062 24619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:35.622290 24626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:35.625775 24626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:35.631974 24626 catalog_manager.cc:1347] Generated new cluster ID: bd8e8f9e0df549c79a6f247766ba443c
I20260810 23:01:35.632061 24626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:35.646234 24626 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:35.647253 24626 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:35.652839 24626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89edf14b10a644358ebd7ce29736a522: Generated new TSK 0
I20260810 23:01:35.653571 24626 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:01:35.689448 24667 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 23:01:35.689640 24667 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:01:35.690208 24612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:35.665244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.15.56.62:36857)
0810 23:01:35.665411 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:35.665415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:35.665576 (+ 161us) server_negotiation.cc:407] Connection header received
0810 23:01:35.665816 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:35.665818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:35.665868 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:35.665954 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:01:35.666222 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:35.666774 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:35.668030 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:35.668260 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:35.668350 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:35.668714 (+ 364us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:01:35.689621 (+ 20907us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:01:35.690022 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38328: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 23:01:35.690320 24612 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38328: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 23:01:35.691736 24665 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:35.665113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:36857 (local address 127.0.0.1:38328)
0810 23:01:35.665491 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:35.665506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:35.665648 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:35.665988 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:35.665991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:35.666001 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:01:35.666126 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:35.666133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:35.666873 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:35.666877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:35.667912 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:35.667924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:35.668378 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:35.668383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:35.668473 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:35.668534 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:01:35.689961 (+ 21427us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:01:35.690233 (+ 272us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.56.62:36857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":124,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.56.62:36857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36297/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 23:01:35.692597 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 24490
2026-08-10T23:01:35Z chronyd exiting
I20260810 23:01:35.742339 15584 test_util.cc:175] -----------------------------------------------
I20260810 23:01:35.742445 15584 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402867637343-15584-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (348 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 23:01:35.743649 15584 mini_oidc.cc:152] Starting JWKS server
I20260810 23:01:35.744591 15584 webserver.cc:460] Webserver started at https://127.0.0.1:46399/ using document root <none> and password file <none>
I20260810 23:01:35.744680 15584 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:01:35.744944 15584 webserver.cc:460] Webserver started at http://127.0.0.1:42953/ using document root <none> and password file <none>
2026-08-10T23:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:35Z Disabled control of system clock
I20260810 23:01:35.756412 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:42907
--webserver_interface=127.15.56.62
--webserver_port=0
--builtin_ntp_servers=127.15.56.20:38983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:42907
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46399/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/testchainca.cert with env {}
W20260810 23:01:35.874068 24680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:35.874389 24680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:35.874432 24680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:35.878619 24680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:35.878712 24680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:35.878727 24680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:35.878743 24680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:35.878757 24680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:01:35.878773 24680 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:01:35.878785 24680 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46399/jwks.json
I20260810 23:01:35.884712 24680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:42907
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:42907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46399/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:35.885957 24680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:35.887120 24680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:35.896364 24690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:35.896373 24691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:35.896574 24680 server_base.cc:1034] running on GCE node
W20260810 23:01:35.896389 24698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:35.897434 24680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:35.898661 24680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:35.899838 24680 hybrid_clock.cc:648] HybridClock initialized: now 1786402895899818 us; error 30 us; skew 500 ppm
I20260810 23:01:35.901903 24680 webserver.cc:460] Webserver started at http://127.15.56.62:40521/ using document root <none> and password file <none>
I20260810 23:01:35.902534 24680 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:35.902614 24680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:35.902832 24680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:35.905102 24680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "bc31a0a1196543d9a4020afbc3a19e91"
format_stamp: "Formatted at 2026-08-10 23:01:35 on dist-test-slave-6bbx"
server_key: "1a27ee2fb4247d19ab8830a7cdb18fb1"
server_key_iv: "a6dd65dcec8bddb6d5a505ce1198506f"
server_key_version: "encryptionkey@0"
I20260810 23:01:35.905684 24680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "bc31a0a1196543d9a4020afbc3a19e91"
format_stamp: "Formatted at 2026-08-10 23:01:35 on dist-test-slave-6bbx"
server_key: "1a27ee2fb4247d19ab8830a7cdb18fb1"
server_key_iv: "a6dd65dcec8bddb6d5a505ce1198506f"
server_key_version: "encryptionkey@0"
I20260810 23:01:35.909564 24680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:01:35.912081 24710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:35.913455 24680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:01:35.913576 24680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "bc31a0a1196543d9a4020afbc3a19e91"
format_stamp: "Formatted at 2026-08-10 23:01:35 on dist-test-slave-6bbx"
server_key: "1a27ee2fb4247d19ab8830a7cdb18fb1"
server_key_iv: "a6dd65dcec8bddb6d5a505ce1198506f"
server_key_version: "encryptionkey@0"
I20260810 23:01:35.913681 24680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:35.946519 24680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:35.947337 24680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:35.947539 24680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:35.954874 24680 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:42907
I20260810 23:01:35.954888 24813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:42907 every 8 connection(s)
I20260810 23:01:35.956117 24680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:35.959693 24814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:35.964262 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 24680
I20260810 23:01:35.964375 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:35.964648 15584 external_mini_cluster.cc:1428] Setting key 300dc4059e0e573381a21a8de79ba59b
I20260810 23:01:35.965843 24814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91: Bootstrap starting.
I20260810 23:01:35.968381 24814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:35.969519 24814 log.cc:824] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:35.972096 24814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91: No bootstrap required, opened a new log
I20260810 23:01:35.974998 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:35.966134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.15.56.62:42907)
0810 23:01:35.966805 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:35.966822 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:35.966862 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:01:35.967543 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:35.967563 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:35.968091 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:35.968468 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:35.968977 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:35.970082 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:35.971382 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:35.971962 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:35.972106 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:35.973039 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:35.973045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:35.973061 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:35.973511 (+ 450us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:35.973571 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:35.973592 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:35.973673 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:35.973880 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 23:01:35.974202 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260810 23:01:35.976140 24814 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31a0a1196543d9a4020afbc3a19e91" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42907 } }
I20260810 23:01:35.976444 24814 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:35.976503 24814 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc31a0a1196543d9a4020afbc3a19e91, State: Initialized, Role: FOLLOWER
I20260810 23:01:35.976989 24814 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31a0a1196543d9a4020afbc3a19e91" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42907 } }
I20260810 23:01:35.977165 24814 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:35.977293 24814 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:35.977412 24814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:35.978302 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:01:35.978614 24814 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31a0a1196543d9a4020afbc3a19e91" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42907 } }
I20260810 23:01:35.978974 24814 leader_election.cc:302] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc31a0a1196543d9a4020afbc3a19e91; no voters:
I20260810 23:01:35.979337 24814 leader_election.cc:288] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:35.979449 24825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:35.979698 24825 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [term 1 LEADER]: Becoming Leader. State: Replica: bc31a0a1196543d9a4020afbc3a19e91, State: Running, Role: LEADER
I20260810 23:01:35.980082 24825 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31a0a1196543d9a4020afbc3a19e91" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42907 } }
I20260810 23:01:35.980727 24814 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:35.981981 24829 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc31a0a1196543d9a4020afbc3a19e91. Latest consensus state: current_term: 1 leader_uuid: "bc31a0a1196543d9a4020afbc3a19e91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31a0a1196543d9a4020afbc3a19e91" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42907 } } }
I20260810 23:01:35.981977 24830 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc31a0a1196543d9a4020afbc3a19e91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc31a0a1196543d9a4020afbc3a19e91" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42907 } } }
I20260810 23:01:35.982189 24829 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:35.982249 24830 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:35.982666 24843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:35.985949 24843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:35.991698 24843 catalog_manager.cc:1347] Generated new cluster ID: b33b26ca0d6540dca05b003440a5c5f1
I20260810 23:01:35.991775 24843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:36.021118 24843 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:36.022123 24843 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:36.032179 24843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc31a0a1196543d9a4020afbc3a19e91: Generated new TSK 0
I20260810 23:01:36.032914 24843 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:01:36.059737 24869 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 23:01:36.059860 24869 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:01:36.060704 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:36.043641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.15.56.62:42907)
0810 23:01:36.043789 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:36.043794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:36.043926 (+ 132us) server_negotiation.cc:407] Connection header received
0810 23:01:36.044083 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:36.044086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:36.044158 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:36.044277 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:01:36.044497 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.045341 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.046307 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.046663 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.046769 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:36.047092 (+ 323us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:01:36.060203 (+ 13111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:01:36.060536 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 23:01:36.060794 24818 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37200: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 23:01:36.060858 24867 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:36.043547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:42907 (local address 127.0.0.1:37200)
0810 23:01:36.043838 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:36.043852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:36.043953 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:36.044292 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:36.044295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:36.044302 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:01:36.044405 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:36.044410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.045499 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.045503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:36.046177 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:36.046189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.046799 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.046802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:36.046875 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:36.046901 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:01:36.060485 (+ 13584us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:01:36.060672 (+ 187us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.56.62:42907: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46399/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":117,"threads_started":1}
I20260810 23:01:36.061592 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 24680
2026-08-10T23:01:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (372 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 23:01:36.116510 15584 mini_oidc.cc:152] Starting JWKS server
I20260810 23:01:36.117400 15584 webserver.cc:460] Webserver started at https://127.0.0.1:39293/ using document root <none> and password file <none>
I20260810 23:01:36.117489 15584 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:01:36.117712 15584 webserver.cc:460] Webserver started at http://127.0.0.1:39061/ using document root <none> and password file <none>
2026-08-10T23:01:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:36Z Disabled control of system clock
I20260810 23:01:36.127434 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:35399
--webserver_interface=127.15.56.62
--webserver_port=0
--builtin_ntp_servers=127.15.56.20:38157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:35399
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39293/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:01:36.243611 24881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:36.243964 24881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:36.244006 24881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:36.248139 24881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:36.248200 24881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:36.248221 24881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:36.248238 24881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:36.248252 24881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:01:36.248267 24881 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:01:36.248279 24881 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39293/jwks.json
I20260810 23:01:36.253157 24881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:35399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:35399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39293/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:36.254258 24881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:36.255683 24881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:36.264346 24897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:36.264539 24881 server_base.cc:1034] running on GCE node
W20260810 23:01:36.264348 24893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:36.264340 24889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:36.265337 24881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:36.266827 24881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:36.268014 24881 hybrid_clock.cc:648] HybridClock initialized: now 1786402896267972 us; error 64 us; skew 500 ppm
I20260810 23:01:36.270213 24881 webserver.cc:460] Webserver started at http://127.15.56.62:38985/ using document root <none> and password file <none>
I20260810 23:01:36.270853 24881 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:36.270929 24881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:36.271152 24881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:36.273391 24881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "262670e983624d2ab74792e1ea16aa30"
format_stamp: "Formatted at 2026-08-10 23:01:36 on dist-test-slave-6bbx"
server_key: "a92bbb5fc0ae6039ab8c98c0c867d03c"
server_key_iv: "a06f6b16f990bbbda2ec2098380d41cc"
server_key_version: "encryptionkey@0"
I20260810 23:01:36.273947 24881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "262670e983624d2ab74792e1ea16aa30"
format_stamp: "Formatted at 2026-08-10 23:01:36 on dist-test-slave-6bbx"
server_key: "a92bbb5fc0ae6039ab8c98c0c867d03c"
server_key_iv: "a06f6b16f990bbbda2ec2098380d41cc"
server_key_version: "encryptionkey@0"
I20260810 23:01:36.277871 24881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:01:36.280673 24908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:36.282116 24881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:01:36.282271 24881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "262670e983624d2ab74792e1ea16aa30"
format_stamp: "Formatted at 2026-08-10 23:01:36 on dist-test-slave-6bbx"
server_key: "a92bbb5fc0ae6039ab8c98c0c867d03c"
server_key_iv: "a06f6b16f990bbbda2ec2098380d41cc"
server_key_version: "encryptionkey@0"
I20260810 23:01:36.282393 24881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:36.301133 24881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:36.301915 24881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:36.302122 24881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:36.308905 24881 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:35399
I20260810 23:01:36.308916 25003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:35399 every 8 connection(s)
I20260810 23:01:36.309918 24881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:36.313205 25004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:36.314301 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 24881
I20260810 23:01:36.314425 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:36.314761 15584 external_mini_cluster.cc:1428] Setting key 83019175ea844a1381a6b2eae24dfa16
I20260810 23:01:36.319689 25004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30: Bootstrap starting.
I20260810 23:01:36.322167 25004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:36.322959 25004 log.cc:824] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:36.324487 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:36.316295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.15.56.62:35399)
0810 23:01:36.316985 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:36.317000 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:36.317037 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:01:36.317687 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:36.317705 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:36.318220 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:36.318543 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:36.319015 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.320075 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.320960 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.321455 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.321607 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:36.322588 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:36.322593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:36.322616 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:36.323027 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:36.323104 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:36.323134 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:36.323226 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:36.323411 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 23:01:36.323726 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
I20260810 23:01:36.325059 25004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30: No bootstrap required, opened a new log
I20260810 23:01:36.327769 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:01:36.329357 25004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262670e983624d2ab74792e1ea16aa30" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35399 } }
I20260810 23:01:36.329676 25004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:36.329737 25004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 262670e983624d2ab74792e1ea16aa30, State: Initialized, Role: FOLLOWER
I20260810 23:01:36.330163 25004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262670e983624d2ab74792e1ea16aa30" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35399 } }
I20260810 23:01:36.330292 25004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:36.330348 25004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:36.330422 25004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:36.331599 25004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262670e983624d2ab74792e1ea16aa30" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35399 } }
I20260810 23:01:36.331962 25004 leader_election.cc:302] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 262670e983624d2ab74792e1ea16aa30; no voters:
I20260810 23:01:36.332268 25004 leader_election.cc:288] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:36.332381 25010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:36.332587 25010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [term 1 LEADER]: Becoming Leader. State: Replica: 262670e983624d2ab74792e1ea16aa30, State: Running, Role: LEADER
I20260810 23:01:36.332931 25010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262670e983624d2ab74792e1ea16aa30" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35399 } }
I20260810 23:01:36.333555 25004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:36.334681 25013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "262670e983624d2ab74792e1ea16aa30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262670e983624d2ab74792e1ea16aa30" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35399 } } }
I20260810 23:01:36.334812 25013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:36.335048 25011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 262670e983624d2ab74792e1ea16aa30. Latest consensus state: current_term: 1 leader_uuid: "262670e983624d2ab74792e1ea16aa30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262670e983624d2ab74792e1ea16aa30" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35399 } } }
I20260810 23:01:36.335161 25011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:36.335299 25025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:36.338330 25025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:36.344194 25025 catalog_manager.cc:1347] Generated new cluster ID: e8d05971ea374680bfd9f24764b0951e
I20260810 23:01:36.344321 25025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:36.362469 25025 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:36.363521 25025 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:36.368721 25025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 262670e983624d2ab74792e1ea16aa30: Generated new TSK 0
I20260810 23:01:36.369436 25025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:01:36.383100 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:36.378217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.15.56.62:35399)
0810 23:01:36.378327 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:36.378331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:36.378561 (+ 230us) server_negotiation.cc:407] Connection header received
0810 23:01:36.378731 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:36.378733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:36.378801 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:36.378913 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:01:36.379158 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.379754 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.380627 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.380863 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.380953 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:36.381280 (+ 327us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:01:36.382688 (+ 1408us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39293/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:01:36.382940 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37790: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39293/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 23:01:36.383176 25007 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37790: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39293/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 23:01:36.385072 25048 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:36.378188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:35399 (local address 127.0.0.1:37790)
0810 23:01:36.378475 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:36.378487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:36.378598 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:36.378925 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:36.378928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:36.378937 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:01:36.379054 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:36.379060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.379869 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.379873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:36.380510 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:36.380518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.380985 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.380989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:36.381079 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:36.381103 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:01:36.382874 (+ 1771us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39293/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:01:36.384860 (+ 1986us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.56.62:35399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39293/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":1728,"thread_start_us":113,"threads_started":1}
I20260810 23:01:36.385926 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 24881
E20260810 23:01:36.398242 25051 webserver.cc:533] Webserver: sslize failed:
E20260810 23:01:36.398344 25051 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T23:01:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T23:01:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:36Z Disabled control of system clock
I20260810 23:01:36.453413 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:42691
--webserver_interface=127.15.56.62
--webserver_port=0
--builtin_ntp_servers=127.15.56.20:42373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:42691
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/testchainca.cert with env {}
W20260810 23:01:36.571873 25061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:36.572199 25061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:36.572243 25061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:36.576419 25061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:36.576490 25061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:36.576504 25061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:36.576520 25061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:36.576534 25061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:01:36.576550 25061 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:01:36.576578 25061 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 23:01:36.582360 25061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:42373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:42691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:42691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:36.583603 25061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:36.584754 25061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:36.593904 25070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:36.593907 25072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:36.593907 25074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:36.594159 25061 server_base.cc:1034] running on GCE node
I20260810 23:01:36.594877 25061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:36.596124 25061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:36.597298 25061 hybrid_clock.cc:648] HybridClock initialized: now 1786402896597287 us; error 29 us; skew 500 ppm
I20260810 23:01:36.599159 25061 webserver.cc:460] Webserver started at http://127.15.56.62:39597/ using document root <none> and password file <none>
I20260810 23:01:36.599759 25061 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:36.599831 25061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:36.600034 25061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:36.602083 25061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "65a24a6069ed4f51a13724993dd26c2d"
format_stamp: "Formatted at 2026-08-10 23:01:36 on dist-test-slave-6bbx"
server_key: "59c78d3848d8c81db99c78fcd82c8f03"
server_key_iv: "e8aee75a4170b40c264f643c0c27b2b4"
server_key_version: "encryptionkey@0"
I20260810 23:01:36.602624 25061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "65a24a6069ed4f51a13724993dd26c2d"
format_stamp: "Formatted at 2026-08-10 23:01:36 on dist-test-slave-6bbx"
server_key: "59c78d3848d8c81db99c78fcd82c8f03"
server_key_iv: "e8aee75a4170b40c264f643c0c27b2b4"
server_key_version: "encryptionkey@0"
I20260810 23:01:36.606264 25061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 23:01:36.608779 25087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:36.610100 25061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:01:36.610253 25061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "65a24a6069ed4f51a13724993dd26c2d"
format_stamp: "Formatted at 2026-08-10 23:01:36 on dist-test-slave-6bbx"
server_key: "59c78d3848d8c81db99c78fcd82c8f03"
server_key_iv: "e8aee75a4170b40c264f643c0c27b2b4"
server_key_version: "encryptionkey@0"
I20260810 23:01:36.610364 25061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:36.647107 25061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:36.648015 25061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:36.648219 25061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:36.655265 25061 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:42691
I20260810 23:01:36.655301 25179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:42691 every 8 connection(s)
I20260810 23:01:36.656380 25061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:36.659432 25182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:36.661264 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 25061
I20260810 23:01:36.661376 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:36.661638 15584 external_mini_cluster.cc:1428] Setting key 73eda71262f2e23793b652d6f206a529
I20260810 23:01:36.666072 25182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d: Bootstrap starting.
I20260810 23:01:36.668792 25182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:36.669653 25182 log.cc:824] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:36.671172 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:36.662988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.15.56.62:42691)
0810 23:01:36.663624 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:36.663641 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:36.663709 (+ 68us) server_negotiation.cc:407] Connection header received
0810 23:01:36.664371 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:36.664390 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:36.664796 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:36.665076 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:36.665468 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.666489 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.667652 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.668174 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.668290 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:36.669221 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:36.669227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:36.669244 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:01:36.669755 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:36.669831 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:01:36.669857 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:36.669951 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:36.670117 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:01:36.670415 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":132,"threads_started":1}
I20260810 23:01:36.671878 25182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d: No bootstrap required, opened a new log
I20260810 23:01:36.674059 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:01:36.675827 25182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a24a6069ed4f51a13724993dd26c2d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42691 } }
I20260810 23:01:36.676047 25182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:36.676100 25182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a24a6069ed4f51a13724993dd26c2d, State: Initialized, Role: FOLLOWER
I20260810 23:01:36.676519 25182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a24a6069ed4f51a13724993dd26c2d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42691 } }
I20260810 23:01:36.676649 25182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:36.676712 25182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:36.676805 25182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:36.677703 25182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a24a6069ed4f51a13724993dd26c2d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42691 } }
I20260810 23:01:36.678033 25182 leader_election.cc:302] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65a24a6069ed4f51a13724993dd26c2d; no voters:
I20260810 23:01:36.678371 25182 leader_election.cc:288] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:36.678470 25191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:36.678697 25191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [term 1 LEADER]: Becoming Leader. State: Replica: 65a24a6069ed4f51a13724993dd26c2d, State: Running, Role: LEADER
I20260810 23:01:36.679013 25191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a24a6069ed4f51a13724993dd26c2d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42691 } }
I20260810 23:01:36.679549 25182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:36.680658 25194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65a24a6069ed4f51a13724993dd26c2d. Latest consensus state: current_term: 1 leader_uuid: "65a24a6069ed4f51a13724993dd26c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a24a6069ed4f51a13724993dd26c2d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42691 } } }
I20260810 23:01:36.680819 25194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:36.681102 25192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65a24a6069ed4f51a13724993dd26c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a24a6069ed4f51a13724993dd26c2d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 42691 } } }
I20260810 23:01:36.681203 25192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:36.681332 25208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:36.684391 25208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:36.690500 25208 catalog_manager.cc:1347] Generated new cluster ID: 90766753af8e420c930e07d0f699a3e9
I20260810 23:01:36.690600 25208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:36.701427 25208 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:36.702661 25208 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:36.707898 25208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65a24a6069ed4f51a13724993dd26c2d: Generated new TSK 0
I20260810 23:01:36.708753 25208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:01:36.773221 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:36.719634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.15.56.62:42691)
0810 23:01:36.719809 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:36.719814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:36.719985 (+ 171us) server_negotiation.cc:407] Connection header received
0810 23:01:36.720157 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:36.720160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:36.720210 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:36.720307 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:01:36.720547 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.721118 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.722441 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.722681 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.722767 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:36.723125 (+ 358us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:01:36.772811 (+ 49686us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:01:36.773095 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33068: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 23:01:36.773319 25187 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33068: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 23:01:36.773854 25236 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:36.719544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:42691 (local address 127.0.0.1:33068)
0810 23:01:36.719897 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:36.719914 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:36.720036 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:36.720325 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:36.720328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:36.720336 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:01:36.720455 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:36.720462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.721231 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.721235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:36.722296 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:36.722311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:36.722818 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:36.722823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:36.722931 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:36.722963 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:01:36.773110 (+ 50147us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:01:36.773346 (+ 236us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.56.62:42691: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
I20260810 23:01:36.774613 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 25061
2026-08-10T23:01:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (360 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (129 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (131 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:37 dist-test-slave-6bbx krb5kdc[25257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:37 dist-test-slave-6bbx krb5kdc[25257](info): set up 2 sockets
Aug 10 23:01:37 dist-test-slave-6bbx krb5kdc[25257](info): commencing operation
krb5kdc: starting...
W20260810 23:01:38.191735 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:38 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:38.323709 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:46697
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:46697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:38.439285 25281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:38.439571 25281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:38.439611 25281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:38.443506 25281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:38.443558 25281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:38.443576 25281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:38.443593 25281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:38.443605 25281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:38.448889 25281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:46697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:38.449947 25281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:38.451018 25281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:38.459331 25291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:38.459331 25297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:38.459342 25292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:38.459532 25281 server_base.cc:1034] running on GCE node
I20260810 23:01:38.460261 25281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:38.461431 25281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:38.462594 25281 hybrid_clock.cc:648] HybridClock initialized: now 1786402898462574 us; error 32 us; skew 500 ppm
Aug 10 23:01:38 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402898, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:38.465433 25281 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:38.466382 25281 webserver.cc:460] Webserver started at http://127.15.56.62:39871/ using document root <none> and password file <none>
I20260810 23:01:38.466943 25281 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:38.467008 25281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:38.467248 25281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:38.469432 25281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "d5446a31da1c49bcb26e68244b1e1cf6"
format_stamp: "Formatted at 2026-08-10 23:01:38 on dist-test-slave-6bbx"
server_key: "42e9d59f99b985579a1f7fd336815dff"
server_key_iv: "bf18ae4651d7da40dd9c753f1ee726fb"
server_key_version: "encryptionkey@0"
I20260810 23:01:38.470065 25281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "d5446a31da1c49bcb26e68244b1e1cf6"
format_stamp: "Formatted at 2026-08-10 23:01:38 on dist-test-slave-6bbx"
server_key: "42e9d59f99b985579a1f7fd336815dff"
server_key_iv: "bf18ae4651d7da40dd9c753f1ee726fb"
server_key_version: "encryptionkey@0"
I20260810 23:01:38.473918 25281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 23:01:38.476356 25311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:38.477569 25281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:38.477687 25281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "d5446a31da1c49bcb26e68244b1e1cf6"
format_stamp: "Formatted at 2026-08-10 23:01:38 on dist-test-slave-6bbx"
server_key: "42e9d59f99b985579a1f7fd336815dff"
server_key_iv: "bf18ae4651d7da40dd9c753f1ee726fb"
server_key_version: "encryptionkey@0"
I20260810 23:01:38.477798 25281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:38.497994 25281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:38.501142 25281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:38.501328 25281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:38.508256 25281 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:46697
I20260810 23:01:38.508277 25411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:46697 every 8 connection(s)
I20260810 23:01:38.509225 25281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:38.511145 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 25281
I20260810 23:01:38.511274 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:38.511590 15584 external_mini_cluster.cc:1428] Setting key 68c3ffb5b393af7db03555f91cab77d5
I20260810 23:01:38.512461 25413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:38.519414 25413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6: Bootstrap starting.
I20260810 23:01:38.521715 25413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:38.522462 25413 log.cc:824] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:38.524780 25413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6: No bootstrap required, opened a new log
Aug 10 23:01:38 dist-test-slave-6bbx krb5kdc[25257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:38.528582 25413 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5446a31da1c49bcb26e68244b1e1cf6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46697 } }
I20260810 23:01:38.528846 25413 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:38.528909 25413 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5446a31da1c49bcb26e68244b1e1cf6, State: Initialized, Role: FOLLOWER
I20260810 23:01:38.529422 25413 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5446a31da1c49bcb26e68244b1e1cf6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46697 } }
I20260810 23:01:38.529595 25413 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:38.529675 25413 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:38.529776 25413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:38.530864 25413 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5446a31da1c49bcb26e68244b1e1cf6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46697 } }
I20260810 23:01:38.531229 25413 leader_election.cc:302] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5446a31da1c49bcb26e68244b1e1cf6; no voters:
I20260810 23:01:38.531353 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:38.513005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.15.56.62:46697)
0810 23:01:38.513426 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:38.513446 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:38.513489 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:01:38.514326 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:38.514350 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:38.514765 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:38.515154 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:38.515864 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:38.517317 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:38.518154 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:38.518834 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:38.519007 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:38.525541 (+ 6534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:38.525548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:38.525565 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:38.525597 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:38.528028 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:38.528447 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:38.528453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:38.528458 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:38.528604 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:38.529020 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:38.529024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:38.529026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:38.529545 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:38.529757 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:38.530025 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 23:01:38.530332 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20260810 23:01:38.531575 25413 leader_election.cc:288] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:38.531752 25419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:38.531981 25419 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [term 1 LEADER]: Becoming Leader. State: Replica: d5446a31da1c49bcb26e68244b1e1cf6, State: Running, Role: LEADER
I20260810 23:01:38.532346 25419 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5446a31da1c49bcb26e68244b1e1cf6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46697 } }
I20260810 23:01:38.532917 25413 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:38.533485 25420 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5446a31da1c49bcb26e68244b1e1cf6. Latest consensus state: current_term: 1 leader_uuid: "d5446a31da1c49bcb26e68244b1e1cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5446a31da1c49bcb26e68244b1e1cf6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46697 } } }
I20260810 23:01:38.533598 25420 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:38.534291 25423 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5446a31da1c49bcb26e68244b1e1cf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5446a31da1c49bcb26e68244b1e1cf6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46697 } } }
I20260810 23:01:38.534385 25423 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:38.534755 25426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:38.537628 25426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:38.542832 25426 catalog_manager.cc:1347] Generated new cluster ID: 399ed077e00b4fbf8a7a2e398cf06d8c
I20260810 23:01:38.542912 25426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:38.565317 25426 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:38.566699 25426 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:38.586074 25426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5446a31da1c49bcb26e68244b1e1cf6: Generated new TSK 0
I20260810 23:01:38.586874 25426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:01:38.622313 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46697
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:38.751822 25451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:38.752104 25451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:38.752148 25451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:38.756148 25451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:38.756201 25451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:38.756258 25451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:38.761533 25451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:38.762580 25451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:38.763561 25451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:38.773293 25463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:38.773293 25468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:38.773524 25451 server_base.cc:1034] running on GCE node
W20260810 23:01:38.773263 25465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:38.774073 25451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:38.774701 25451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:38.775861 25451 hybrid_clock.cc:648] HybridClock initialized: now 1786402898775836 us; error 35 us; skew 500 ppm
Aug 10 23:01:38 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402898, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:38.778482 25451 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:01:38.779304 25451 webserver.cc:460] Webserver started at http://127.15.56.1:41917/ using document root <none> and password file <none>
I20260810 23:01:38.779872 25451 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:38.779942 25451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:38.780144 25451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:38.782145 25451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "422af72af4534bbf878737994bd7d5d1"
format_stamp: "Formatted at 2026-08-10 23:01:38 on dist-test-slave-6bbx"
server_key: "4dad974929fbe8591ea39f750d77bd45"
server_key_iv: "3d1724a348c0e929c54b2fae69dadc96"
server_key_version: "encryptionkey@0"
I20260810 23:01:38.782675 25451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "422af72af4534bbf878737994bd7d5d1"
format_stamp: "Formatted at 2026-08-10 23:01:38 on dist-test-slave-6bbx"
server_key: "4dad974929fbe8591ea39f750d77bd45"
server_key_iv: "3d1724a348c0e929c54b2fae69dadc96"
server_key_version: "encryptionkey@0"
I20260810 23:01:38.786213 25451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:01:38.788870 25483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:38.790197 25451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:38.790343 25451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "422af72af4534bbf878737994bd7d5d1"
format_stamp: "Formatted at 2026-08-10 23:01:38 on dist-test-slave-6bbx"
server_key: "4dad974929fbe8591ea39f750d77bd45"
server_key_iv: "3d1724a348c0e929c54b2fae69dadc96"
server_key_version: "encryptionkey@0"
I20260810 23:01:38.790446 25451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:38.807265 25451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:38.810385 25451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:38.810573 25451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:38.811169 25451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:38.812033 25451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:38.812083 25451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:38.812146 25451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:38.812173 25451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:38.820521 25451 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:42211
I20260810 23:01:38.820549 25702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:42211 every 8 connection(s)
I20260810 23:01:38.821408 25451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:38.830183 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 25451
I20260810 23:01:38.830310 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:38.830574 15584 external_mini_cluster.cc:1428] Setting key 6787bd6303d1c2733489b55f275d976f
Aug 10 23:01:38 dist-test-slave-6bbx krb5kdc[25257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402898, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:38.835253 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:38.823369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:37549 (local address 127.15.56.62:46697)
0810 23:01:38.823550 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:38.823555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:38.824490 (+ 935us) server_negotiation.cc:407] Connection header received
0810 23:01:38.825828 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:38.825830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:38.825878 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:38.825953 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:38.827142 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:38.827727 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:38.828857 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:38.829095 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:38.829196 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:38.831743 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:38.831746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:38.831747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:38.831773 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:38.833179 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:38.833676 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:38.833680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:38.833682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:38.833760 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:38.834124 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:38.834127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:38.834129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:38.834349 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:38.834437 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:38.834959 (+ 522us) server_negotiation.cc:299] Negotiation successful
0810 23:01:38.835091 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:01:38.836303 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:38.823800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:46697 (local address 127.15.56.1:37549)
0810 23:01:38.824306 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:38.824356 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:38.825627 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:38.826111 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:38.826118 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:38.826504 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:38.827008 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:38.827023 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:38.827838 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:38.827841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:38.828723 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:38.828733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:38.829264 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:38.829267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:38.829416 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:38.829989 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:38.830006 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:38.831552 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:38.833288 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:38.833294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:38.833309 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:38.833592 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:38.833893 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:38.833896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:38.833899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:38.834035 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:38.834448 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:38.834455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:38.834731 (+ 276us) client_negotiation.cc:769] Sending connection context
0810 23:01:38.834905 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 23:01:38.835240 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":134,"threads_started":1}
I20260810 23:01:38.837559 25706 heartbeater.cc:347] Connected to a master server at 127.15.56.62:46697
I20260810 23:01:38.837797 25706 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:38.838307 25706 heartbeater.cc:511] Master 127.15.56.62:46697 requested a full tablet report, sending...
I20260810 23:01:38.839915 25342 ts_manager.cc:194] Registered new tserver with Master: 422af72af4534bbf878737994bd7d5d1 (127.15.56.1:42211)
I20260810 23:01:38.841037 25342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:37549
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:01:38.858480 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46697
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:38.973596 25713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:38.973876 25713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:38.973937 25713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:38.978017 25713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:38.978081 25713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:38.978171 25713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:38.983412 25713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:38.984467 25713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:38.985493 25713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:38.995065 25724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:38.995082 25722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:38.995065 25721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:38.995296 25713 server_base.cc:1034] running on GCE node
I20260810 23:01:38.995764 25713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:38.996397 25713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:38.997556 25713 hybrid_clock.cc:648] HybridClock initialized: now 1786402898997525 us; error 42 us; skew 500 ppm
Aug 10 23:01:38 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402898, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:39.000253 25713 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:01:39.001075 25713 webserver.cc:460] Webserver started at http://127.15.56.2:38997/ using document root <none> and password file <none>
I20260810 23:01:39.001606 25713 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:39.001665 25713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:39.001864 25713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:39.003938 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "32ac9f2af5b640cabc241252903c3092"
format_stamp: "Formatted at 2026-08-10 23:01:38 on dist-test-slave-6bbx"
server_key: "6f16b4f22f4ea5111a59d8b809a1027a"
server_key_iv: "5caed55399bfd463993aa1077494a72b"
server_key_version: "encryptionkey@0"
I20260810 23:01:39.004477 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "32ac9f2af5b640cabc241252903c3092"
format_stamp: "Formatted at 2026-08-10 23:01:38 on dist-test-slave-6bbx"
server_key: "6f16b4f22f4ea5111a59d8b809a1027a"
server_key_iv: "5caed55399bfd463993aa1077494a72b"
server_key_version: "encryptionkey@0"
I20260810 23:01:39.008271 25713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:01:39.010917 25734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:39.012331 25713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:39.012439 25713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "32ac9f2af5b640cabc241252903c3092"
format_stamp: "Formatted at 2026-08-10 23:01:38 on dist-test-slave-6bbx"
server_key: "6f16b4f22f4ea5111a59d8b809a1027a"
server_key_iv: "5caed55399bfd463993aa1077494a72b"
server_key_version: "encryptionkey@0"
I20260810 23:01:39.012548 25713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:39.057946 25713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:39.061120 25713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:39.061291 25713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:39.061913 25713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:39.062780 25713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:39.062836 25713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:39.062883 25713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:39.062913 25713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:39.071640 25713 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:44759
I20260810 23:01:39.071712 25949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:44759 every 8 connection(s)
I20260810 23:01:39.072649 25713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:39.076122 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 25713
I20260810 23:01:39.076360 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:39.076730 15584 external_mini_cluster.cc:1428] Setting key 453c9ed805648f3b3073f292238b2850
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402898, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:01:39.089172 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:39.074736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:36335 (local address 127.15.56.62:46697)
0810 23:01:39.074880 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:39.074884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:39.076123 (+ 1239us) server_negotiation.cc:407] Connection header received
0810 23:01:39.077814 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:39.077817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:39.077875 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:39.077950 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:39.079795 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.080552 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.082059 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.082311 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.082406 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:39.085203 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:39.085207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:39.085209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:39.085237 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:39.086750 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:39.087390 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:39.087393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:39.087394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:39.087479 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:39.087926 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:39.087928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:39.087930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:39.088120 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:39.088247 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:39.088891 (+ 644us) server_negotiation.cc:299] Negotiation successful
0810 23:01:39.089004 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":36}
I20260810 23:01:39.089986 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:39.075309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:46697 (local address 127.15.56.2:36335)
0810 23:01:39.075937 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:39.076003 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:39.077588 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:39.078318 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:39.078329 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:39.078869 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:39.079600 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:39.079623 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.080735 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.080739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:39.081912 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:39.081923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.082467 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.082471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:39.082659 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:39.083258 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:39.083275 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:39.084999 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:39.086877 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:39.086884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:39.086905 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:39.087267 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:39.087572 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:39.087576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:39.087578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:39.087817 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:39.088238 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:39.088245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:39.088578 (+ 333us) client_negotiation.cc:769] Sending connection context
0810 23:01:39.088766 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 23:01:39.089107 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
I20260810 23:01:39.091361 25951 heartbeater.cc:347] Connected to a master server at 127.15.56.62:46697
I20260810 23:01:39.091624 25951 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:39.092182 25951 heartbeater.cc:511] Master 127.15.56.62:46697 requested a full tablet report, sending...
I20260810 23:01:39.093344 25342 ts_manager.cc:194] Registered new tserver with Master: 32ac9f2af5b640cabc241252903c3092 (127.15.56.2:44759)
I20260810 23:01:39.093990 25342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:36335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:01:39.107187 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46697
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:39.226642 25957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:39.226960 25957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:39.227007 25957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:39.231226 25957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:39.231310 25957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:39.231371 25957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:39.237205 25957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:39.238406 25957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:39.239548 25957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:39.250564 25969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:39.250564 25963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:39.250737 25957 server_base.cc:1034] running on GCE node
W20260810 23:01:39.250564 25965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:39.251425 25957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:39.252199 25957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:39.253396 25957 hybrid_clock.cc:648] HybridClock initialized: now 1786402899253359 us; error 58 us; skew 500 ppm
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:39.256505 25957 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:01:39.257579 25957 webserver.cc:460] Webserver started at http://127.15.56.3:41287/ using document root <none> and password file <none>
I20260810 23:01:39.258237 25957 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:39.258318 25957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:39.258533 25957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:39.260877 25957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "2d22347481044b34902c5046d9b7dc0c"
format_stamp: "Formatted at 2026-08-10 23:01:39 on dist-test-slave-6bbx"
server_key: "37c42df67debdb5fa5c43e3a02bc8bf9"
server_key_iv: "43ffb2ab4b1dbea38f880e82f03bbbda"
server_key_version: "encryptionkey@0"
I20260810 23:01:39.261480 25957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "2d22347481044b34902c5046d9b7dc0c"
format_stamp: "Formatted at 2026-08-10 23:01:39 on dist-test-slave-6bbx"
server_key: "37c42df67debdb5fa5c43e3a02bc8bf9"
server_key_iv: "43ffb2ab4b1dbea38f880e82f03bbbda"
server_key_version: "encryptionkey@0"
I20260810 23:01:39.265740 25957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:01:39.268424 25987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:39.269786 25957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:39.269899 25957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "2d22347481044b34902c5046d9b7dc0c"
format_stamp: "Formatted at 2026-08-10 23:01:39 on dist-test-slave-6bbx"
server_key: "37c42df67debdb5fa5c43e3a02bc8bf9"
server_key_iv: "43ffb2ab4b1dbea38f880e82f03bbbda"
server_key_version: "encryptionkey@0"
I20260810 23:01:39.270021 25957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:39.297353 25957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:39.301023 25957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:39.301218 25957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:39.301879 25957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:39.302829 25957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:39.302881 25957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:39.302923 25957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:39.302953 25957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:39.311793 25957 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:33541
I20260810 23:01:39.311841 26171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:33541 every 8 connection(s)
I20260810 23:01:39.312798 25957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:39.314884 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 25957
I20260810 23:01:39.315006 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:39.315292 15584 external_mini_cluster.cc:1428] Setting key 1dee07dc57c1f1758fee14102896a1d3
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:39.327556 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:39.314823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:48281 (local address 127.15.56.62:46697)
0810 23:01:39.314990 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:39.314996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:39.316161 (+ 1165us) server_negotiation.cc:407] Connection header received
0810 23:01:39.317929 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:39.317932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:39.317989 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:39.318067 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:39.319370 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.319964 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.321140 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.321365 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.321458 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:39.324047 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:39.324050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:39.324052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:39.324079 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:39.325323 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:39.325865 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:39.325868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:39.325869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:39.325959 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:39.326386 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:39.326390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:39.326391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:39.326599 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:39.326697 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:39.327292 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 23:01:39.327415 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:01:39.328555 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:39.315370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:46697 (local address 127.15.56.3:48281)
0810 23:01:39.315989 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:39.316057 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:39.317703 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:39.318285 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:39.318292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:39.318700 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:39.319226 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:39.319241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.320095 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.320098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:39.321019 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:39.321029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.321513 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.321517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:39.321703 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:39.322309 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:39.322326 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:39.323871 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:39.325447 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:39.325453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:39.325467 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:39.325745 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:39.326075 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:39.326078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:39.326080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:39.326267 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:39.326734 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:39.326740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:39.327020 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 23:01:39.327233 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 23:01:39.327533 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":161,"threads_started":1}
I20260810 23:01:39.329733 26173 heartbeater.cc:347] Connected to a master server at 127.15.56.62:46697
I20260810 23:01:39.329957 26173 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:39.330433 26173 heartbeater.cc:511] Master 127.15.56.62:46697 requested a full tablet report, sending...
I20260810 23:01:39.331449 25340 ts_manager.cc:194] Registered new tserver with Master: 2d22347481044b34902c5046d9b7dc0c (127.15.56.3:33541)
I20260810 23:01:39.332067 25340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:48281
I20260810 23:01:39.341517 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/test-admin.keytab.
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:39.360874 15584 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 23:01:39.364358 26188 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:39.362599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.15.56.1:42211)
0810 23:01:39.363019 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:39.363027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:39.363053 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:01:39.363106 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:39.363111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:39.363338 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:39.363473 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:39.364197 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55904: BlockingRecv error: recv got EOF from 127.0.0.1:55904 (error 108)
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":193,"threads_started":1}
W20260810 23:01:39.364475 15584 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:39.365657 15584 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:39.366695 15584 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:01:39.377209 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:39.367773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.15.56.1:42211)
0810 23:01:39.367919 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:39.367922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:39.367953 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:01:39.368116 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:39.368119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:39.368198 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:39.368275 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:39.368828 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.369645 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.370657 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.371077 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.371178 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:39.373669 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:39.373686 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:39.373706 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:39.373737 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:39.375477 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:39.375965 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:39.375971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:39.375972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:39.376057 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:39.376269 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:39.376272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:39.376273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:39.376540 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:39.376706 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:39.376867 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 23:01:39.377023 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 23:01:39.381651 26188 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:39.380554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.15.56.1:42211)
0810 23:01:39.380699 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:39.380703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:39.380776 (+ 73us) server_negotiation.cc:407] Connection header received
0810 23:01:39.380901 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:39.380904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:39.380949 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:39.381009 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:39.381522 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55926: BlockingRecv error: recv got EOF from 127.0.0.1:55926 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 23:01:39.381722 15584 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:39.382827 15584 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:39.383699 15584 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:01:39.392432 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:39.384589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.15.56.1:42211)
0810 23:01:39.384745 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:39.384748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:39.384783 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:01:39.384896 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:39.384898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:39.384938 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:39.385003 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:39.385488 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.386007 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.386801 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.387028 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.387117 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:39.389314 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:39.389317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:39.389320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:39.389347 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:39.390741 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:39.391182 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:39.391186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:39.391188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:39.391270 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:39.391485 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:39.391487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:39.391488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:39.391712 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:39.391844 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:39.392142 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 23:01:39.392263 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 23:01:39.395385 26188 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:39.394347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.15.56.1:42211)
0810 23:01:39.394486 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:39.394490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:39.394586 (+ 96us) server_negotiation.cc:407] Connection header received
0810 23:01:39.394704 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:39.394707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:39.394751 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:39.394817 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:39.395269 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55942: BlockingRecv error: recv got EOF from 127.0.0.1:55942 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 23:01:39.395488 15584 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402867637343-15584-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:39.396539 15584 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:39.397553 15584 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[25257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:01:39.406088 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:39.398577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.15.56.1:42211)
0810 23:01:39.398709 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:39.398714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:39.398812 (+ 98us) server_negotiation.cc:407] Connection header received
0810 23:01:39.398916 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:39.398919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:39.398959 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:39.399028 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:39.399557 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.400112 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.400896 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:39.401114 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:39.401197 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:39.403431 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:39.403434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:39.403436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:39.403461 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:39.404618 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:39.405011 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:39.405014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:39.405016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:39.405097 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:39.405343 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:39.405346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:39.405348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:39.405537 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:39.405672 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:39.405823 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 23:01:39.405925 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:01:39.407171 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 25451
I20260810 23:01:39.434506 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 25713
I20260810 23:01:39.449097 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 25957
I20260810 23:01:39.464591 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 25281
2026-08-10T23:01:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2429 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[26208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[26208](info): set up 2 sockets
Aug 10 23:01:39 dist-test-slave-6bbx krb5kdc[26208](info): commencing operation
krb5kdc: starting...
W20260810 23:01:40.628203 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:40 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.62.keytab.
Entry for principal oryx/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.62.keytab.
I20260810 23:01:40.762648 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:40785
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.62.keytab
--principal=oryx/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:40785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:40.880020 26228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:40.880304 26228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:40.880344 26228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:40.884325 26228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:40.884382 26228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:40.884399 26228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:40.884415 26228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:40.884429 26228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:40.889636 26228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:40785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.62.keytab
--principal=oryx/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:40785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:40.890735 26228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:40.891837 26228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:40.900583 26237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:40.900573 26239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:40.900573 26236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:40.900986 26228 server_base.cc:1034] running on GCE node
I20260810 23:01:40.901659 26228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:40.902916 26228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:40.904064 26228 hybrid_clock.cc:648] HybridClock initialized: now 1786402900904040 us; error 42 us; skew 500 ppm
Aug 10 23:01:40 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402900, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:40.910322 26228 init.cc:377] Logged in from keytab as oryx/127.15.56.62@KRBTEST.COM (short username oryx)
I20260810 23:01:40.911417 26228 webserver.cc:460] Webserver started at http://127.15.56.62:37589/ using document root <none> and password file <none>
I20260810 23:01:40.912086 26228 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:40.912170 26228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:40.912412 26228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:40.914568 26228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "0f99da2aa72a42b798e738c6a97b4c14"
format_stamp: "Formatted at 2026-08-10 23:01:40 on dist-test-slave-6bbx"
server_key: "d3b2137d795cfd711225c0583c727a8c"
server_key_iv: "145d7bd8bd4a4c8553372bf58d90c14e"
server_key_version: "encryptionkey@0"
I20260810 23:01:40.915277 26228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "0f99da2aa72a42b798e738c6a97b4c14"
format_stamp: "Formatted at 2026-08-10 23:01:40 on dist-test-slave-6bbx"
server_key: "d3b2137d795cfd711225c0583c727a8c"
server_key_iv: "145d7bd8bd4a4c8553372bf58d90c14e"
server_key_version: "encryptionkey@0"
I20260810 23:01:40.919253 26228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 23:01:40.921711 26256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:40.922946 26228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:40.923064 26228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "0f99da2aa72a42b798e738c6a97b4c14"
format_stamp: "Formatted at 2026-08-10 23:01:40 on dist-test-slave-6bbx"
server_key: "d3b2137d795cfd711225c0583c727a8c"
server_key_iv: "145d7bd8bd4a4c8553372bf58d90c14e"
server_key_version: "encryptionkey@0"
I20260810 23:01:40.923164 26228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:40.940977 26228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:40.944237 26228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:40.944433 26228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:40.951586 26228 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:40785
I20260810 23:01:40.951879 26340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:40785 every 8 connection(s)
I20260810 23:01:40.952672 26228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:40.955703 26341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:40.960732 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 26228
I20260810 23:01:40.960886 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:40.961200 15584 external_mini_cluster.cc:1428] Setting key f99839575376d75b380fea72165850a6
I20260810 23:01:40.962410 26341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14: Bootstrap starting.
I20260810 23:01:40.964985 26341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:40.965935 26341 log.cc:824] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:40.968258 26341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14: No bootstrap required, opened a new log
Aug 10 23:01:40 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:40.973011 26341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } }
I20260810 23:01:40.973344 26341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:40.973418 26341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f99da2aa72a42b798e738c6a97b4c14, State: Initialized, Role: FOLLOWER
I20260810 23:01:40.973961 26341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } }
I20260810 23:01:40.974143 26341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:40.974222 26341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:40.974332 26341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:40.975559 26341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } }
I20260810 23:01:40.975987 26341 leader_election.cc:302] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f99da2aa72a42b798e738c6a97b4c14; no voters:
I20260810 23:01:40.976354 26341 leader_election.cc:288] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:40.976512 26347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:40.976748 26347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 1 LEADER]: Becoming Leader. State: Replica: 0f99da2aa72a42b798e738c6a97b4c14, State: Running, Role: LEADER
I20260810 23:01:40.976976 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:40.962771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.15.56.62:40785)
0810 23:01:40.963221 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:40.963237 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:40.963275 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:01:40.963993 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:40.964014 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:40.964378 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:40.964688 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:40.965644 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:40.966695 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:40.967853 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:40.968359 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:40.968528 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:40.971323 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:40.971332 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:40.971356 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:40.971399 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:40.973385 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:40.973882 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:40.973887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:40.973894 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:40.974047 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:40.974391 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:40.974395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:40.974396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:40.974950 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:40.975154 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:40.975498 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 23:01:40.975893 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":143,"threads_started":1}
I20260810 23:01:40.977154 26347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } }
I20260810 23:01:40.977700 26341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:40.979095 26350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f99da2aa72a42b798e738c6a97b4c14. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } }
I20260810 23:01:40.979095 26355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } }
I20260810 23:01:40.979260 26350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:40.979260 26355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:40.979837 26370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:40.983254 26370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:40.989102 26370 catalog_manager.cc:1347] Generated new cluster ID: 2dff39177dc5487bb114ba9d5d63b58a
I20260810 23:01:40.989167 26370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:41.005275 26370 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:41.006228 26370 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:41.014493 26370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14: Generated new TSK 0
I20260810 23:01:41.015203 26370 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab.
Entry for principal oryx/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab.
I20260810 23:01:41.050405 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40785
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab
--principal=oryx/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:41.172288 26381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:41.172590 26381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:41.172638 26381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:41.176733 26381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:41.176800 26381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:41.176862 26381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:41.182268 26381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab
--principal=oryx/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:41.183379 26381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:41.184486 26381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:41.193957 26387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:41.194020 26381 server_base.cc:1034] running on GCE node
W20260810 23:01:41.193953 26398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:41.193953 26392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:41.194554 26381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:41.195206 26381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:41.196388 26381 hybrid_clock.cc:648] HybridClock initialized: now 1786402901196353 us; error 45 us; skew 500 ppm
Aug 10 23:01:41 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402901, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:41.199158 26381 init.cc:377] Logged in from keytab as oryx/127.15.56.1@KRBTEST.COM (short username oryx)
I20260810 23:01:41.200038 26381 webserver.cc:460] Webserver started at http://127.15.56.1:40019/ using document root <none> and password file <none>
I20260810 23:01:41.200551 26381 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:41.200608 26381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:41.200783 26381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:41.202745 26381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "55a97d9983524ec28f5aef1a5176efb3"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "df59499d5381c7bf6804b73f7501643d"
server_key_iv: "5b86f41c11ca1902ef2ea08f9e06c428"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.203284 26381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "55a97d9983524ec28f5aef1a5176efb3"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "df59499d5381c7bf6804b73f7501643d"
server_key_iv: "5b86f41c11ca1902ef2ea08f9e06c428"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.206924 26381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:01:41.209290 26409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:41.210489 26381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:01:41.210585 26381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "55a97d9983524ec28f5aef1a5176efb3"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "df59499d5381c7bf6804b73f7501643d"
server_key_iv: "5b86f41c11ca1902ef2ea08f9e06c428"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.210666 26381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:41.229957 26381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:41.233268 26381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:41.233459 26381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:41.234076 26381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:41.234934 26381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:41.234985 26381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:41.235039 26381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:41.235067 26381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:41.243695 26381 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:40683
I20260810 23:01:41.243750 26584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:40683 every 8 connection(s)
I20260810 23:01:41.244609 26381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:41.247906 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 26381
I20260810 23:01:41.248025 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:41.248323 15584 external_mini_cluster.cc:1428] Setting key f57363b779abed95422e9d155f2b4e17
Aug 10 23:01:41 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402901, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:01:41.258661 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.246448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:52197 (local address 127.15.56.62:40785)
0810 23:01:41.246626 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:41.246630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:41.247460 (+ 830us) server_negotiation.cc:407] Connection header received
0810 23:01:41.248961 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:41.248965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:41.249024 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:41.249097 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:41.250438 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.250972 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.252562 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.252808 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.252894 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.255260 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:41.255263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:41.255265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:41.255291 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:41.256674 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.257157 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.257160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.257161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.257243 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.257559 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.257562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.257564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.257744 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:41.257840 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:41.258357 (+ 517us) server_negotiation.cc:299] Negotiation successful
0810 23:01:41.258490 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 23:01:41.259585 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.246829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.1:52197)
0810 23:01:41.247302 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:41.247348 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:41.248728 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:41.249297 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:41.249305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:41.249769 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:41.250293 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.250308 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.251092 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.251095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.252434 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.252447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.252957 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.252961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.253119 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.253677 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:41.253694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:41.255096 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:41.256787 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.256793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.256810 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.257063 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.257329 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.257332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.257334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.257473 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.257853 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:41.257860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:41.258122 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 23:01:41.258309 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 23:01:41.258641 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.2@KRBTEST.COM" created.
I20260810 23:01:41.260946 26585 heartbeater.cc:347] Connected to a master server at 127.15.56.62:40785
I20260810 23:01:41.261204 26585 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:41.261691 26585 heartbeater.cc:511] Master 127.15.56.62:40785 requested a full tablet report, sending...
I20260810 23:01:41.263242 26288 ts_manager.cc:194] Registered new tserver with Master: 55a97d9983524ec28f5aef1a5176efb3 (127.15.56.1:40683)
I20260810 23:01:41.264439 26288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:52197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab.
Entry for principal oryx/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab.
I20260810 23:01:41.276933 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40785
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab
--principal=oryx/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:41.394186 26592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:41.394450 26592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:41.394497 26592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:41.398434 26592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:41.398490 26592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:41.398547 26592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:41.404022 26592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab
--principal=oryx/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:41.405107 26592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:41.406123 26592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:41.416455 26608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:41.416417 26604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:41.416561 26592 server_base.cc:1034] running on GCE node
W20260810 23:01:41.416440 26605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:41.417248 26592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:41.417999 26592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:41.419185 26592 hybrid_clock.cc:648] HybridClock initialized: now 1786402901419144 us; error 55 us; skew 500 ppm
Aug 10 23:01:41 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402901, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:41.422432 26592 init.cc:377] Logged in from keytab as oryx/127.15.56.2@KRBTEST.COM (short username oryx)
I20260810 23:01:41.423496 26592 webserver.cc:460] Webserver started at http://127.15.56.2:43827/ using document root <none> and password file <none>
I20260810 23:01:41.424239 26592 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:41.424330 26592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:41.424562 26592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:41.426772 26592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "166e4b15b20145908fb57ace1c03ce7f"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "b3b0a57a1d7f149c2fb86745701a0371"
server_key_iv: "1aee984f59573a0cbbe2836b28403a38"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.427414 26592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "166e4b15b20145908fb57ace1c03ce7f"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "b3b0a57a1d7f149c2fb86745701a0371"
server_key_iv: "1aee984f59573a0cbbe2836b28403a38"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.431526 26592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260810 23:01:41.434182 26626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:41.435521 26592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:41.435645 26592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "166e4b15b20145908fb57ace1c03ce7f"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "b3b0a57a1d7f149c2fb86745701a0371"
server_key_iv: "1aee984f59573a0cbbe2836b28403a38"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.435787 26592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:41.451562 26592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:41.454694 26592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:41.454888 26592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:41.455547 26592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:41.456504 26592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:41.456558 26592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:41.456615 26592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:41.456645 26592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:41.465636 26592 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:34583
I20260810 23:01:41.465677 26799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:34583 every 8 connection(s)
I20260810 23:01:41.466609 26592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:41.474936 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 26592
I20260810 23:01:41.475066 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:41.475383 15584 external_mini_cluster.cc:1428] Setting key 999a8f5037553eb605924d6f5a30295b
Aug 10 23:01:41 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402901, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:41.482421 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.468581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:47297 (local address 127.15.56.62:40785)
0810 23:01:41.468738 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:41.468744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:41.469813 (+ 1069us) server_negotiation.cc:407] Connection header received
0810 23:01:41.471257 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:41.471260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:41.471339 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:41.471421 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:41.472897 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.473500 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.474924 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.475248 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.475350 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.478460 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:41.478464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:41.478466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:41.478492 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:41.480241 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.480780 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.480784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.480786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.480876 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.481186 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.481189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.481191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.481440 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:41.481543 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:41.482088 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 23:01:41.482246 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:01:41.483332 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.469074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.2:47297)
0810 23:01:41.469622 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:41.469673 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:41.470991 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:41.471606 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:41.471614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:41.472109 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:41.472682 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.472699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.473684 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.473687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.474752 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.474767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.475448 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.475454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.475778 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.476453 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:41.476472 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:41.478254 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:41.480368 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.480373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.480390 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.480681 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.480950 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.480952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.480953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.481100 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.481534 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:41.481539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:41.481837 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 23:01:41.482010 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 23:01:41.482328 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":141,"threads_started":1}
I20260810 23:01:41.484675 26801 heartbeater.cc:347] Connected to a master server at 127.15.56.62:40785
I20260810 23:01:41.484988 26801 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:41.485637 26801 heartbeater.cc:511] Master 127.15.56.62:40785 requested a full tablet report, sending...
I20260810 23:01:41.486886 26288 ts_manager.cc:194] Registered new tserver with Master: 166e4b15b20145908fb57ace1c03ce7f (127.15.56.2:34583)
WARNING: no policy specified for oryx/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:01:41.487531 26288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:47297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab.
Entry for principal oryx/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab.
I20260810 23:01:41.506961 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40785
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab
--principal=oryx/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:41.621486 26814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:41.621750 26814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:41.621793 26814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:41.625787 26814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:41.625845 26814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:41.625902 26814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:41.631162 26814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab
--principal=oryx/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:41.632217 26814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:41.633230 26814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:41.643065 26823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:41.643167 26825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:41.643087 26828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:41.643141 26814 server_base.cc:1034] running on GCE node
I20260810 23:01:41.643788 26814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:41.644425 26814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:41.645579 26814 hybrid_clock.cc:648] HybridClock initialized: now 1786402901645551 us; error 40 us; skew 500 ppm
Aug 10 23:01:41 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402901, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:41.648294 26814 init.cc:377] Logged in from keytab as oryx/127.15.56.3@KRBTEST.COM (short username oryx)
I20260810 23:01:41.649202 26814 webserver.cc:460] Webserver started at http://127.15.56.3:38799/ using document root <none> and password file <none>
I20260810 23:01:41.649758 26814 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:41.649825 26814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:41.650032 26814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:41.652172 26814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "14835bd4d1074a1194177fdf78deacab"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "0ba9a0f5b5290f400779fd3aab6d8e7b"
server_key_iv: "791650d58af80afd78164831d57bdcde"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.652726 26814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "14835bd4d1074a1194177fdf78deacab"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "0ba9a0f5b5290f400779fd3aab6d8e7b"
server_key_iv: "791650d58af80afd78164831d57bdcde"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.656597 26814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:01:41.659044 26841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:41.660351 26814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:01:41.660475 26814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "14835bd4d1074a1194177fdf78deacab"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "0ba9a0f5b5290f400779fd3aab6d8e7b"
server_key_iv: "791650d58af80afd78164831d57bdcde"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.660573 26814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:41.685190 26814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:41.688585 26814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:41.688779 26814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:41.689430 26814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:41.690305 26814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:41.690356 26814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:41.690410 26814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:41.690438 26814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:41.699033 26814 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:45637
I20260810 23:01:41.699062 27038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:45637 every 8 connection(s)
I20260810 23:01:41.700004 26814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:41.704258 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 26814
I20260810 23:01:41.704377 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:41.704643 15584 external_mini_cluster.cc:1428] Setting key 21838adf9f03256a2d53d7108147a451
Aug 10 23:01:41 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402901, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:41.713724 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.701848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:36715 (local address 127.15.56.62:40785)
0810 23:01:41.702028 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:41.702033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:41.702933 (+ 900us) server_negotiation.cc:407] Connection header received
0810 23:01:41.704310 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:41.704313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:41.704375 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:41.704455 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:41.706006 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.706534 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.707718 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.707958 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.708051 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.710504 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:41.710507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:41.710509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:41.710534 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:41.711690 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.712179 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.712182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.712185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.712285 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.712603 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.712606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.712608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.712808 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:41.712895 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:41.713427 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 23:01:41.713558 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:01:41.714581 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.702304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.3:36715)
0810 23:01:41.702765 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:41.702824 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:41.704082 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:41.704722 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:41.704731 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:41.705195 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:41.705858 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.705882 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.706658 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.706662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.707575 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.707586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.708100 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.708105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.708296 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.708841 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:41.708858 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:41.710340 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:41.711795 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.711801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.711817 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.712082 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.712370 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.712373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.712375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.712515 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.712912 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:41.712919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:41.713186 (+ 267us) client_negotiation.cc:769] Sending connection context
0810 23:01:41.713369 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 23:01:41.713705 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
I20260810 23:01:41.715767 27039 heartbeater.cc:347] Connected to a master server at 127.15.56.62:40785
I20260810 23:01:41.715996 27039 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:41.716480 27039 heartbeater.cc:511] Master 127.15.56.62:40785 requested a full tablet report, sending...
I20260810 23:01:41.717553 26288 ts_manager.cc:194] Registered new tserver with Master: 14835bd4d1074a1194177fdf78deacab (127.15.56.3:45637)
I20260810 23:01:41.718240 26288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:36715
I20260810 23:01:41.718642 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.56.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.61.keytab.
Entry for principal oryx/127.15.56.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.61.keytab.
I20260810 23:01:41.747282 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.61:39989
--webserver_interface=127.15.56.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.61.keytab
--principal=oryx/127.15.56.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:40785,127.15.56.61:39989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:41.862952 27051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:41.863229 27051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:41.863286 27051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:41.867287 27051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:41.867355 27051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:41.867383 27051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:41.867403 27051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:41.867424 27051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:41.872712 27051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:40785,127.15.56.61:39989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.61.keytab
--principal=oryx/127.15.56.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.61:39989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.56.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
W20260810 23:01:41.872951 27051 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.56.62:40785,127.15.56.61:39989'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 23:01:41.873826 27051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:41.874861 27051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:41.883535 27058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:41.883574 27057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:41.883701 27060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:41.883802 27051 server_base.cc:1034] running on GCE node
I20260810 23:01:41.884343 27051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:41.885372 27051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:41.886533 27051 hybrid_clock.cc:648] HybridClock initialized: now 1786402901886512 us; error 38 us; skew 500 ppm
Aug 10 23:01:41 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402901, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:41.889354 27051 init.cc:377] Logged in from keytab as oryx/127.15.56.61@KRBTEST.COM (short username oryx)
I20260810 23:01:41.890269 27051 webserver.cc:460] Webserver started at http://127.15.56.61:44347/ using document root <none> and password file <none>
I20260810 23:01:41.890811 27051 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:41.890873 27051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:41.891073 27051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:41.893136 27051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data/instance:
uuid: "78fe7d9db0874fccbd42624a4e89c37f"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "685be683b77b9ea29ccacb5da240ed84"
server_key_iv: "60386e07b51f451acdefc19459676db6"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.893668 27051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/wal/instance:
uuid: "78fe7d9db0874fccbd42624a4e89c37f"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "685be683b77b9ea29ccacb5da240ed84"
server_key_iv: "60386e07b51f451acdefc19459676db6"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.897578 27051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:01:41.900102 27068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:41.901377 27051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:01:41.901495 27051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/wal
uuid: "78fe7d9db0874fccbd42624a4e89c37f"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "685be683b77b9ea29ccacb5da240ed84"
server_key_iv: "60386e07b51f451acdefc19459676db6"
server_key_version: "encryptionkey@0"
I20260810 23:01:41.901593 27051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:41.914254 27051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:41.917156 27051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:41.917346 27051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:41.924183 27051 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.61:39989
I20260810 23:01:41.924464 27168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.61:39989 every 8 connection(s)
I20260810 23:01:41.925158 27051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
I20260810 23:01:41.928104 27175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:41.931878 27175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:01:41.934557 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 27051
Aug 10 23:01:41 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402901, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.61@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
Aug 10 23:01:41 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.56.61@KRBTEST.COM
I20260810 23:01:41.946481 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.933766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.15.56.62:40785)
0810 23:01:41.933904 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:41.933908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:41.934759 (+ 851us) server_negotiation.cc:407] Connection header received
0810 23:01:41.935885 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:41.935888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:41.935945 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:41.936022 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:41.937238 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.938060 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.939512 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.939929 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.940037 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.942894 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:41.942898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:41.942901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:41.942933 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:41.944475 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.944990 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.944993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.944995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.945075 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.945403 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.945406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.945407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.945646 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:41.945740 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:41.946190 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 23:01:41.946314 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:01:41.947374 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.934132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:45556)
0810 23:01:41.934606 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:41.934658 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:41.935515 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:41.936219 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:41.936226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:41.936604 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:41.937070 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.937090 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.938213 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.938216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.939398 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.939410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.940098 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.940101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.940284 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.940973 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:41.940994 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:41.942717 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:41.944579 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.944588 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.944608 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.944892 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.945167 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.945170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.945172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.945309 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.945753 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:41.945760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:41.945968 (+ 208us) client_negotiation.cc:769] Sending connection context
0810 23:01:41.946122 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 23:01:41.946448 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":46,"thread_start_us":154,"threads_started":1}
I20260810 23:01:41.947378 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.935537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.15.56.61:39989)
0810 23:01:41.936115 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:41.936127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:41.936148 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:01:41.936195 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:41.936198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:41.936373 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:41.936506 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:41.937344 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.938476 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.939574 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.940135 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.940335 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.943266 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:41.943273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:41.943289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:41.943310 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:41.944817 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.945208 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.945212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.945214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.945318 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.945585 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.945588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.945590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.945958 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:41.946152 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:41.946359 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 23:01:41.946497 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":487,"spinlock_wait_cycles":4608,"thread_start_us":147,"threads_started":1}
I20260810 23:01:41.948809 27175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:01:41 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402901, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.61@KRBTEST.COM for oryx/127.15.56.61@KRBTEST.COM
I20260810 23:01:41.956964 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.949266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:39989 (local address 127.0.0.1:33506)
0810 23:01:41.949377 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:41.949390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:41.949461 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:41.949778 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:41.949780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:41.949910 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:41.950002 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.950007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.950710 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.950712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.951299 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.951311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.952374 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.952378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.952477 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.952899 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:41.952912 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:41.954175 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:41.955610 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.955613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.955615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.955842 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.956079 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.956082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.956084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.956159 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.956484 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:41.956487 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:41.956570 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 23:01:41.956653 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:01:41.956791 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 23:01:41.956967 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.949504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.15.56.61:39989)
0810 23:01:41.949607 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:41.949610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:41.949621 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:01:41.949646 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:41.949648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:41.949680 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:41.949737 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:41.950082 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.950588 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.951425 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.952256 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.952346 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.954292 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:41.954296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:41.954299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:41.954320 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:41.955512 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.955932 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.955936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.955937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.956004 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.956234 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.956237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.956239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.956400 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:41.956484 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:41.956705 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 23:01:41.956846 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":38}
I20260810 23:01:41.959754 27175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: Bootstrap starting.
I20260810 23:01:41.962040 27175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:41.962785 27175 log.cc:824] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:41.964799 27175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: No bootstrap required, opened a new log
I20260810 23:01:41.967813 27175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } }
I20260810 23:01:41.968005 27175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:41.968055 27175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78fe7d9db0874fccbd42624a4e89c37f, State: Initialized, Role: FOLLOWER
I20260810 23:01:41.968472 27175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } }
I20260810 23:01:41.968922 27184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } } }
I20260810 23:01:41.969039 27184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:41.969631 27175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:01:41.974063 27206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:01:41.974138 27206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:01:41.976191 27051 master_runner.cc:428] Detected that this master 78fe7d9db0874fccbd42624a4e89c37f is joining an existing cluster
I20260810 23:01:41.976243 27051 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.56.61:39989
I20260810 23:01:41.986644 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.978189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.15.56.62:40785)
0810 23:01:41.978340 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:41.978345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:41.978477 (+ 132us) server_negotiation.cc:407] Connection header received
0810 23:01:41.978602 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:41.978606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:41.978653 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:41.978718 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:41.979436 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.980317 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.981399 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.981625 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.981713 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.982837 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:41.982840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:41.982843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:41.982869 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:41.984397 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.985415 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.985418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.985419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.985492 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.985818 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.985822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.985824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.986049 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:41.986146 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:41.986369 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 23:01:41.986487 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 23:01:41.986644 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.978127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:45566)
0810 23:01:41.978400 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:41.978414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:41.978522 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:41.978848 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:41.978851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:41.979153 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:41.979312 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.979320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.980414 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.980417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.981276 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.981284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.981733 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.981736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.981837 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.982345 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:41.982360 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:41.982703 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:41.984508 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.984511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.984513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.984766 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.985582 (+ 816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.985585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.985587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.985669 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.986141 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:41.986143 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:41.986214 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 23:01:41.986308 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 23:01:41.986446 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":20,"thread_start_us":100,"threads_started":1}
I20260810 23:01:41.986775 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.978884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.15.56.61:39989)
0810 23:01:41.978980 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:41.978985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:41.978997 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:01:41.979031 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:41.979034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:41.979073 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:41.979153 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:41.979528 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.980181 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.981352 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.981697 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.981809 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.983487 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:41.983491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:41.983494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:41.983519 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:41.985310 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.985722 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.985725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.985727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.985800 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.986022 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.986025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.986026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.986212 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:41.986313 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:41.986570 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 23:01:41.986663 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":34}
I20260810 23:01:41.987056 27213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.978571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:39989 (local address 127.0.0.1:33516)
0810 23:01:41.978861 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:41.978873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:41.978949 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:41.979166 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:41.979169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:41.979337 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:41.979448 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.979455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.980285 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.980289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.981221 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.981231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.981915 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.981918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.982011 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.982919 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:41.982934 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:41.983349 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:41.985398 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.985402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.985404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.985629 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.985872 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.985875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.985877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.985948 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.986344 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:41.986346 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:41.986420 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 23:01:41.986862 (+ 442us) client_negotiation.cc:240] Negotiation successful
0810 23:01:41.986954 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
Aug 10 23:01:41 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402900, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.62@KRBTEST.COM for oryx/127.15.56.61@KRBTEST.COM
I20260810 23:01:42.000748 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.991553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:39989 (local address 127.0.0.1:33526)
0810 23:01:41.991943 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:41.992004 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:41.992139 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:41.992416 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:41.992422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:41.992875 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:41.993017 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.993025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.993678 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.993681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.994702 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:41.994713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.995119 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.995123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:41.995241 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.995808 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:41.995823 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:41.997329 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:41.998982 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.998992 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.998998 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.999235 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:41.999511 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:41.999514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:41.999516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:41.999623 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.000089 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.000096 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.000226 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.000376 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.000576 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":113,"threads_started":1}
I20260810 23:01:42.000754 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:41.991576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.15.56.61:39989)
0810 23:01:41.991712 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:41.991717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:41.992083 (+ 366us) server_negotiation.cc:407] Connection header received
0810 23:01:41.992264 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:41.992266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:41.992307 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:41.992369 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:41.993111 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.993588 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.994819 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:41.995022 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:41.995098 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:41.997465 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:41.997468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:41.997470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:41.997491 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:41.998867 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.999349 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.999352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.999353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.999426 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:41.999741 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:41.999745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:41.999747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:41.999967 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.000061 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.000488 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.000603 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:01:42.002549 26288 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.56.61:39989: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f"
member_type: NON_VOTER
last_known_addr {
host: "127.15.56.61"
port: 39989
}
attrs {
promote: true
}
}
dest_uuid: "0f99da2aa72a42b798e738c6a97b4c14"
cas_config_opid_index: -1
I20260810 23:01:42.003209 26288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } }
I20260810 23:01:42.004873 27219 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 78fe7d9db0874fccbd42624a4e89c37f (127.15.56.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } } }
I20260810 23:01:42.005720 27109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:42.006680 27219 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.56.61:39989
I20260810 23:01:42.007108 27051 master.cc:537] Master@127.15.56.61:39989 shutting down...
I20260810 23:01:42.007081 27218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } } }
I20260810 23:01:42.007138 27109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Refusing update from remote peer 0f99da2aa72a42b798e738c6a97b4c14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 23:01:42.007174 27218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:42.008031 27218 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:01:42.009899 26257 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 23:01:42.010762 26257 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 -> Peer 78fe7d9db0874fccbd42624a4e89c37f (127.15.56.61:39989): Couldn't send request to peer 78fe7d9db0874fccbd42624a4e89c37f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:01:42.011519 27051 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:42.011837 27051 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:42.011950 27051 tablet_replica.cc:330] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: stopping tablet replica
WARNING: no policy specified for oryx/127.15.56.60@KRBTEST.COM; defaulting to no policy
I20260810 23:01:42.017235 27051 master.cc:552] Master@127.15.56.61:39989 shutdown complete.
I20260810 23:01:42.017316 27051 master_runner.cc:305] Clearing existing system tablet
I20260810 23:01:42.017556 27051 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:42.017616 27051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.60@KRBTEST.COM" created.
I20260810 23:01:42.018945 27051 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:01:42.019819 27224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:42.020052 27051 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:42.020179 27051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:42.020216 27051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/wal
uuid: "78fe7d9db0874fccbd42624a4e89c37f"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "685be683b77b9ea29ccacb5da240ed84"
server_key_iv: "60386e07b51f451acdefc19459676db6"
server_key_version: "encryptionkey@0"
I20260810 23:01:42.022313 27051 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:42.024699 27051 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:01:42.024802 27051 log.cc:1196] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 23:01:42.025054 27051 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: Deleting consensus metadata
I20260810 23:01:42.025213 27051 master_runner.cc:315] Copying system tablet from 127.15.56.62:40785
I20260810 23:01:42.026371 27051 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: tablet copy: Beginning tablet copy session from remote peer at address 127.15.56.62:40785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.60.keytab.
Entry for principal oryx/127.15.56.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.60.keytab.
I20260810 23:01:42.035063 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.026902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:45578)
0810 23:01:42.027272 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.027287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.027401 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.027781 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.027783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.027941 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.028044 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.028050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.029191 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.029193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.029813 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.029822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.030387 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.030390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.030464 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.030998 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.031014 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.031504 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.033351 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.033354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.033356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.033650 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.033938 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.033941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.033943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.034062 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.034465 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.034468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.034579 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.034685 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.034845 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":97,"threads_started":1}
I20260810 23:01:42.035063 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.026994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.15.56.62:40785)
0810 23:01:42.027125 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.027129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.027369 (+ 240us) server_negotiation.cc:407] Connection header received
0810 23:01:42.027510 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.027513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.027566 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.027697 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.028149 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.029075 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.029930 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.030270 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.030408 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.031642 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.031646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.031648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.031817 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.033194 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.033766 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.033769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.033770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.033862 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.034151 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.034154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.034156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.034377 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.034471 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.034762 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.034911 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:01:42.035319 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.60:38129
--webserver_interface=127.15.56.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.60.keytab
--principal=oryx/127.15.56.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:40785,127.15.56.61:39989,127.15.56.60:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:01:42.036120 26318 tablet_copy_service.cc:140] P 0f99da2aa72a42b798e738c6a97b4c14: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 78fe7d9db0874fccbd42624a4e89c37f ({username='oryx', principal='oryx/127.15.56.61@KRBTEST.COM'} at 127.0.0.1:45578)
I20260810 23:01:42.036324 26318 tablet_copy_service.cc:161] P 0f99da2aa72a42b798e738c6a97b4c14: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 78fe7d9db0874fccbd42624a4e89c37f at {username='oryx', principal='oryx/127.15.56.61@KRBTEST.COM'} at 127.0.0.1:45578: session id = 78fe7d9db0874fccbd42624a4e89c37f-00000000000000000000000000000000
I20260810 23:01:42.038178 26318 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:01:42.040129 27051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:01:42.042575 27051 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: tablet copy: Starting download of 0 data blocks...
I20260810 23:01:42.042788 27051 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: tablet copy: Starting download of 1 WAL segments...
I20260810 23:01:42.044744 27051 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:01:42.046646 26318 tablet_copy_service.cc:342] P 0f99da2aa72a42b798e738c6a97b4c14: Request end of tablet copy session 78fe7d9db0874fccbd42624a4e89c37f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.56.61@KRBTEST.COM'} at 127.0.0.1:45578
I20260810 23:01:42.046773 26318 tablet_copy_service.cc:433] P 0f99da2aa72a42b798e738c6a97b4c14: ending tablet copy session 78fe7d9db0874fccbd42624a4e89c37f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 78fe7d9db0874fccbd42624a4e89c37f
I20260810 23:01:42.054952 27051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:42.056675 27238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:42.056871 27051 server_base.cc:1034] running on GCE node
W20260810 23:01:42.060153 27241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:42.060634 27239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:42.060855 27051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:42.061168 27051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:42.062332 27051 hybrid_clock.cc:648] HybridClock initialized: now 1786402902062315 us; error 37 us; skew 500 ppm
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:42.063984 27051 init.cc:377] Logged in from keytab as oryx/127.15.56.61@KRBTEST.COM (short username oryx)
I20260810 23:01:42.064459 27051 webserver.cc:460] Webserver started at http://127.15.56.61:41601/ using document root <none> and password file <none>
I20260810 23:01:42.064663 27051 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:42.064706 27051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:42.066056 27051 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:42.067080 27251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:42.067337 27051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:42.067413 27051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/wal
uuid: "78fe7d9db0874fccbd42624a4e89c37f"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "685be683b77b9ea29ccacb5da240ed84"
server_key_iv: "60386e07b51f451acdefc19459676db6"
server_key_version: "encryptionkey@0"
I20260810 23:01:42.067487 27051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:42.086932 27051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:42.089236 27051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:42.094395 27051 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.61:39989
I20260810 23:01:42.094444 27354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.61:39989 every 8 connection(s)
I20260810 23:01:42.095013 27051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
I20260810 23:01:42.097043 27356 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:01:42.098188 27356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: Bootstrap starting.
I20260810 23:01:42.110062 27356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:01:42.110400 27356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: Bootstrap complete.
I20260810 23:01:42.111006 27356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } }
I20260810 23:01:42.111151 27356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 78fe7d9db0874fccbd42624a4e89c37f, State: Initialized, Role: LEARNER
I20260810 23:01:42.111268 27356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } }
I20260810 23:01:42.111640 27361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } } }
I20260810 23:01:42.111794 27356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:42.111876 27361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: This master's current role is: LEARNER
I20260810 23:01:42.115645 27377 catalog_manager.cc:1259] Loaded cluster ID: 2dff39177dc5487bb114ba9d5d63b58a
I20260810 23:01:42.115748 27377 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: loading cluster ID for follower catalog manager: success
I20260810 23:01:42.117889 27377 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: acquiring CA information for follower catalog manager: success
I20260810 23:01:42.119421 27377 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 23:01:42.170260 27237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:42.170513 27237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:42.170552 27237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:42.174542 27237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:42.174606 27237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:42.174624 27237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:42.174639 27237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:42.174652 27237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:42.180055 27237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:40785,127.15.56.61:39989,127.15.56.60:38129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.60.keytab
--principal=oryx/127.15.56.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.60:38129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.56.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:42.181135 27237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:42.182226 27237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:42.190961 27386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:42.190955 27387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:42.191001 27237 server_base.cc:1034] running on GCE node
W20260810 23:01:42.190955 27391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:42.191859 27237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:42.193030 27237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:42.194200 27237 hybrid_clock.cc:648] HybridClock initialized: now 1786402902194170 us; error 41 us; skew 500 ppm
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:42.197177 27237 init.cc:377] Logged in from keytab as oryx/127.15.56.60@KRBTEST.COM (short username oryx)
I20260810 23:01:42.198105 27237 webserver.cc:460] Webserver started at http://127.15.56.60:39067/ using document root <none> and password file <none>
I20260810 23:01:42.198647 27237 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:42.198705 27237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:42.198890 27237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:42.201094 27237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data/instance:
uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037"
format_stamp: "Formatted at 2026-08-10 23:01:42 on dist-test-slave-6bbx"
server_key: "724139b3ce1b2abd98cb1d3b85a20513"
server_key_iv: "194c3d1f09ed4567128d21acbb531e48"
server_key_version: "encryptionkey@0"
I20260810 23:01:42.201650 27237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/wal/instance:
uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037"
format_stamp: "Formatted at 2026-08-10 23:01:42 on dist-test-slave-6bbx"
server_key: "724139b3ce1b2abd98cb1d3b85a20513"
server_key_iv: "194c3d1f09ed4567128d21acbb531e48"
server_key_version: "encryptionkey@0"
I20260810 23:01:42.205600 27237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:01:42.208029 27401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:42.209249 27237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:01:42.209367 27237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/wal
uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037"
format_stamp: "Formatted at 2026-08-10 23:01:42 on dist-test-slave-6bbx"
server_key: "724139b3ce1b2abd98cb1d3b85a20513"
server_key_iv: "194c3d1f09ed4567128d21acbb531e48"
server_key_version: "encryptionkey@0"
I20260810 23:01:42.209470 27237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:42.231834 27237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:42.235132 27237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:42.235318 27237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:42.242645 27237 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.60:38129
I20260810 23:01:42.242928 27501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.60:38129 every 8 connection(s)
I20260810 23:01:42.243747 27237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
I20260810 23:01:42.246685 27505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:42.250665 27505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:01:42.253355 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 27237
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.60@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.56.60@KRBTEST.COM
I20260810 23:01:42.264077 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.252820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.15.56.62:40785)
0810 23:01:42.252951 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.252955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.253694 (+ 739us) server_negotiation.cc:407] Connection header received
0810 23:01:42.254787 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.254791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.254850 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.254923 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.256238 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.256783 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.257878 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.258092 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.258192 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.260637 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.260640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.260642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.260667 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.262052 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.262607 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.262611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.262613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.262693 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.263020 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.263023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.263025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.263201 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.263295 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.263785 (+ 490us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.263909 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:01:42.264936 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.253003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:45590)
0810 23:01:42.253524 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.253578 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.254540 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.255004 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.255013 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.255480 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.256092 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.256113 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.256911 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.256915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.257751 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.257761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.258235 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.258240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.258391 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.258912 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.258928 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.260488 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.262169 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.262178 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.262189 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.262515 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.262792 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.262795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.262797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.262928 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.263305 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.263312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.263551 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.263734 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.264051 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":6,"spinlock_wait_cycles":7936,"thread_start_us":141,"threads_started":1}
I20260810 23:01:42.264945 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.254388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.15.56.60:38129)
0810 23:01:42.254726 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.254772 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.254812 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:01:42.254974 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.254979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.255198 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.255381 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.256114 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.257077 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.257884 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.258253 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.258397 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.261016 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.261023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.261036 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.261061 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.262554 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.262950 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.262955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.262956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.263031 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.263279 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.263283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.263284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.263541 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.263756 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.263968 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.264109 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":241,"spinlock_wait_cycles":2048,"thread_start_us":147,"threads_started":1}
I20260810 23:01:42.266120 27505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:01:42.267349 26585 heartbeater.cc:503] Master 127.15.56.62:40785 was elected leader, sending a full tablet report...
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.60@KRBTEST.COM for oryx/127.15.56.61@KRBTEST.COM
I20260810 23:01:42.273852 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.266581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:39989 (local address 127.0.0.1:33532)
0810 23:01:42.266742 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.266756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.266836 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.267194 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.267196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.267333 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.267434 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.267441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.268510 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.268513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.269120 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.269129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.269536 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.269538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.269601 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.270002 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.270014 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.271209 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.272694 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.272696 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.272698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.272876 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.273082 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.273084 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.273085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.273148 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.273470 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.273472 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.273552 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.273631 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.273729 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 23:01:42.273902 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.266690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.15.56.61:39989)
0810 23:01:42.266963 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.266968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.266984 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:01:42.267025 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.267028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.267075 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.267188 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.267546 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.268404 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.269215 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.269455 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.269533 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.271360 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.271364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.271366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.271391 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.272601 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.272949 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.272951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.272952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.273022 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.273205 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.273207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.273208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.273386 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.273460 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.273678 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.273777 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":111,"threads_started":1}
I20260810 23:01:42.274544 27505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.60@KRBTEST.COM for oryx/127.15.56.60@KRBTEST.COM
I20260810 23:01:42.281314 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.274985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:38129 (local address 127.0.0.1:47684)
0810 23:01:42.275104 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.275118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.275196 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.275398 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.275401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.275540 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.275601 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.275606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.276290 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.276293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.276784 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.276792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.277190 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.277194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.277275 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.277638 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.277649 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.278738 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.280119 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.280122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.280123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.280342 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.280559 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.280561 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.280562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.280633 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.280903 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.280905 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.280974 (+ 69us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.281043 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.281130 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":2}
I20260810 23:01:42.281322 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.275035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.15.56.60:38129)
0810 23:01:42.275120 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.275124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.275158 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:01:42.275283 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.275285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.275316 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.275373 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.275708 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.276185 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.276882 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.277100 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.277181 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.278875 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.278878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.278880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.278900 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.280025 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.280425 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.280427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.280428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.280492 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.280694 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.280696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.280697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.280841 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.280908 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.281100 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.281187 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 23:01:42.284796 27505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: Bootstrap starting.
I20260810 23:01:42.287505 27505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:42.288522 27505 log.cc:824] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:42.290578 27505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: No bootstrap required, opened a new log
I20260810 23:01:42.293694 27505 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } }
I20260810 23:01:42.293905 27505 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:42.293944 27505 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ebe0dbffb74ffdaaa35b4b39c8f037, State: Initialized, Role: FOLLOWER
I20260810 23:01:42.294345 27505 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } }
I20260810 23:01:42.294796 27515 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } } }
I20260810 23:01:42.294924 27515 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:42.295532 27505 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:01:42.300138 27538 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:01:42.300213 27538 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:01:42.304030 27237 master_runner.cc:428] Detected that this master f7ebe0dbffb74ffdaaa35b4b39c8f037 is joining an existing cluster
I20260810 23:01:42.304098 27237 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.56.60:38129
I20260810 23:01:42.318301 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 26381
I20260810 23:01:42.332006 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.306416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.15.56.60:38129)
0810 23:01:42.306530 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.306535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.307169 (+ 634us) server_negotiation.cc:407] Connection header received
0810 23:01:42.307284 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.307286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.307320 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.307359 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.308209 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.308998 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.310292 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.310660 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.310771 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.312112 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.312117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.312119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.312163 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.329651 (+ 17488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.330331 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.330335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.330337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.330430 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.330948 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.330952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.330954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.331206 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.331282 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.331706 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.331836 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":44}
I20260810 23:01:42.333161 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.306616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:38129 (local address 127.0.0.1:47690)
0810 23:01:42.307094 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.307106 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.307210 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.307468 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.307471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.307853 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.308095 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.308103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.309123 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.309126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.310195 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.310202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.310795 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.310799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.310894 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.311449 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.311466 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.312012 (+ 546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.329784 (+ 17772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.329787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.329789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.330012 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.330524 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.330526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.330528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.330844 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.331337 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.331340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.331447 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.332589 (+ 1142us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.332968 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":179,"spinlock_wait_cycles":5376,"thread_start_us":96,"threads_started":1}
I20260810 23:01:42.335702 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.307171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.15.56.61:39989)
0810 23:01:42.307297 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.307302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.307513 (+ 211us) server_negotiation.cc:407] Connection header received
0810 23:01:42.307718 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.307721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.307789 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.307844 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.308855 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.309650 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.311484 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.311852 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.311964 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.329778 (+ 17814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.329783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.329786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.329821 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.331368 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.334251 (+ 2883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.334255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.334257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.334358 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.334701 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.334705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.334707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.334951 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.335031 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.335387 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.335529 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:01:42.336100 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.306429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.15.56.62:40785)
0810 23:01:42.306536 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.306540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.306984 (+ 444us) server_negotiation.cc:407] Connection header received
0810 23:01:42.307198 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.307202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.307259 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.307324 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.308234 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.309009 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.310290 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.310626 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.310692 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.331930 (+ 21238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.331935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.331938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.331972 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.333463 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.334696 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.334700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.334702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.334811 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.335137 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.335140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.335142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.335388 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.335467 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.335844 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.335963 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:01:42.336622 27546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.306353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:45598)
0810 23:01:42.306895 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.306908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.307071 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.307468 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.307471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.307853 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.308102 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.308109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.309123 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.309126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.310140 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.310151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.310847 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.310850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.310970 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.329096 (+ 18126us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.329115 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.331736 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.333829 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.333832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.333834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.334065 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.334925 (+ 860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.334928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.334929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.335021 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.335539 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.335542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.335630 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.336253 (+ 623us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.336418 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":70,"thread_start_us":132,"threads_started":1}
I20260810 23:01:42.336627 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:40683
--local_ip_for_outbound_sockets=127.15.56.1
--tserver_master_addrs=127.15.56.62:40785
--webserver_port=40019
--webserver_interface=127.15.56.1
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab
--principal=oryx/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.56.62:40785,127.15.56.61:39989
--tserver_master_addrs=127.15.56.62:40785,127.15.56.61:39989,127.15.56.60:38129 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:01:42.337241 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.306832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:39989 (local address 127.0.0.1:33546)
0810 23:01:42.307429 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.307442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.307581 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.307989 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.307992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.308222 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.308693 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.308703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.309806 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.309809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.311011 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.311019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.311971 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.311975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.312072 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.312485 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.312500 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.329540 (+ 17040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.331497 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.331501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.331503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.334148 (+ 2645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.334469 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.334473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.334475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.334556 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.335104 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.335107 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.335210 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.336104 (+ 894us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.336496 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":111,"thread_start_us":89,"threads_started":1}
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402900, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.62@KRBTEST.COM for oryx/127.15.56.60@KRBTEST.COM
I20260810 23:01:42.349891 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.342482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.15.56.60:38129)
0810 23:01:42.342591 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.342595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.342606 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:01:42.342674 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.342676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.342720 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.342784 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.343223 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.343815 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.344710 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.344938 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.345025 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.347008 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.347011 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.347013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.347036 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.348352 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.348750 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.348753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.348754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.348835 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.349085 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.349089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.349090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.349271 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.349359 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.349615 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.349722 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:01:42.349891 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.342302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:38129 (local address 127.0.0.1:47694)
0810 23:01:42.342414 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.342430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.342509 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.342826 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.342829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.343018 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.343111 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.343116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.343924 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.343928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.344596 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.344607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.345041 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.345045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.345139 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.345579 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.345591 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.346872 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.348458 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.348461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.348463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.348661 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.348920 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.348923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.348925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.349007 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.349364 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.349368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.349465 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.349553 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.349711 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 23:01:42.351186 26288 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.56.60:38129: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037"
member_type: NON_VOTER
last_known_addr {
host: "127.15.56.60"
port: 38129
}
attrs {
promote: true
}
}
dest_uuid: "0f99da2aa72a42b798e738c6a97b4c14"
cas_config_opid_index: 5
I20260810 23:01:42.351999 26288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } }
I20260810 23:01:42.353114 27218 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f7ebe0dbffb74ffdaaa35b4b39c8f037 (127.15.56.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.354450 27218 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.56.60:38129
I20260810 23:01:42.354648 27218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.354745 27218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:42.355317 27237 master.cc:537] Master@127.15.56.60:38129 shutting down...
I20260810 23:01:42.355742 27463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:42.357441 27463 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 FOLLOWER]: Refusing update from remote peer 0f99da2aa72a42b798e738c6a97b4c14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 23:01:42.358894 27219 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:01:42.360888 26261 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 -> Peer f7ebe0dbffb74ffdaaa35b4b39c8f037 (127.15.56.60:38129): Couldn't send request to peer f7ebe0dbffb74ffdaaa35b4b39c8f037. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:01:42.362380 27237 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:42.362798 27237 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:42.363003 27237 tablet_replica.cc:330] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: stopping tablet replica
I20260810 23:01:42.366550 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.353525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.15.56.61:39989)
0810 23:01:42.353651 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.353656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.361582 (+ 7926us) server_negotiation.cc:407] Connection header received
0810 23:01:42.362198 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.362202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.362291 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.362368 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:42.362802 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.363554 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.365041 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.365689 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.365773 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.365935 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.366325 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.366414 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:01:42.366968 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.353257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:39989 (local address 127.0.0.1:33548)
0810 23:01:42.361490 (+ 8233us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.361508 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.362011 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.362559 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.362562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.362580 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:42.362673 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.362679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.363720 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.363724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.364888 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.364901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.366034 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.366038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.366131 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.366186 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.366756 (+ 570us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.366844 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8127}
I20260810 23:01:42.367528 27298 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 LEARNER]: Refusing update from remote peer 0f99da2aa72a42b798e738c6a97b4c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 23:01:42.368170 27219 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 23:01:42.370043 27298 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f7ebe0dbffb74ffdaaa35b4b39c8f037 (127.15.56.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.372330 27361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f99da2aa72a42b798e738c6a97b4c14. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.372453 27361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: This master's current role is: LEARNER
I20260810 23:01:42.372553 27361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: NON_VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: true } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.372629 27361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: This master's current role is: LEARNER
I20260810 23:01:42.372721 27237 master.cc:552] Master@127.15.56.60:38129 shutdown complete.
I20260810 23:01:42.372763 27237 master_runner.cc:305] Clearing existing system tablet
I20260810 23:01:42.373014 27237 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:42.373072 27237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:42.373490 27219 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14: attempting to promote NON_VOTER 78fe7d9db0874fccbd42624a4e89c37f to VOTER
I20260810 23:01:42.374009 27219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } }
I20260810 23:01:42.375387 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.375065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:38129 (local address 127.0.0.1:47700)
0810 23:01:42.375214 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.375301 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 23:01:42.375696 27298 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 LEARNER]: Refusing update from remote peer 0f99da2aa72a42b798e738c6a97b4c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 23:01:42.376098 27556 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 23:01:42.376559 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.376316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:38129 (local address 127.0.0.1:47712)
0810 23:01:42.376446 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.376498 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:38129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260810 23:01:42.377108 26261 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 -> Peer f7ebe0dbffb74ffdaaa35b4b39c8f037 (127.15.56.60:38129): Couldn't send request to peer f7ebe0dbffb74ffdaaa35b4b39c8f037. Status: Network error: Client connection negotiation failed: client connection to 127.15.56.60:38129: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:01:42.377333 27237 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 23:01:42.378283 27218 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 78fe7d9db0874fccbd42624a4e89c37f (127.15.56.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.378937 27298 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 78fe7d9db0874fccbd42624a4e89c37f (127.15.56.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.380218 27218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.380337 27218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:42.380486 27218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.380563 27218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:42.380927 27554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.381057 27554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:42.381471 27559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:42.381708 27237 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:42.381856 27237 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260810 23:01:42.381896 27237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/wal
uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037"
format_stamp: "Formatted at 2026-08-10 23:01:42 on dist-test-slave-6bbx"
server_key: "724139b3ce1b2abd98cb1d3b85a20513"
server_key_iv: "194c3d1f09ed4567128d21acbb531e48"
server_key_version: "encryptionkey@0"
I20260810 23:01:42.383852 27237 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:42.385753 27237 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:01:42.385849 27237 log.cc:1196] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 23:01:42.386075 27237 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: Deleting consensus metadata
I20260810 23:01:42.386230 27237 master_runner.cc:315] Copying system tablet from 127.15.56.62:40785
I20260810 23:01:42.387356 27237 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: tablet copy: Beginning tablet copy session from remote peer at address 127.15.56.62:40785
I20260810 23:01:42.394719 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.387978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.15.56.62:40785)
0810 23:01:42.388100 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.388105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.388292 (+ 187us) server_negotiation.cc:407] Connection header received
0810 23:01:42.388426 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.388430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.388478 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.388545 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.389011 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.389535 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.390357 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.390582 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.390669 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.391847 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.391851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.391854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.391880 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.393193 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.393586 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.393589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.393591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.393671 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.393920 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.393924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.393926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.394102 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.394189 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.394456 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.394569 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:01:42.394984 27571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.387911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:45606)
0810 23:01:42.388202 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.388214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.388333 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.388574 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.388577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.388787 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.388895 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.388901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.389638 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.389641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.390236 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.390243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.390685 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.390688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.390781 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.391265 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.391288 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.391640 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.393295 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.393299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.393301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.393495 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.393754 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.393757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.393759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.393840 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.394191 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.394195 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.394300 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.394387 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.394532 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":83,"threads_started":1}
I20260810 23:01:42.395411 26318 tablet_copy_service.cc:140] P 0f99da2aa72a42b798e738c6a97b4c14: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f7ebe0dbffb74ffdaaa35b4b39c8f037 ({username='oryx', principal='oryx/127.15.56.60@KRBTEST.COM'} at 127.0.0.1:45606)
I20260810 23:01:42.395504 26318 tablet_copy_service.cc:161] P 0f99da2aa72a42b798e738c6a97b4c14: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f7ebe0dbffb74ffdaaa35b4b39c8f037 at {username='oryx', principal='oryx/127.15.56.60@KRBTEST.COM'} at 127.0.0.1:45606: session id = f7ebe0dbffb74ffdaaa35b4b39c8f037-00000000000000000000000000000000
I20260810 23:01:42.396449 26318 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:01:42.397629 27237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:01:42.399412 27237 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: tablet copy: Starting download of 0 data blocks...
I20260810 23:01:42.399590 27237 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: tablet copy: Starting download of 1 WAL segments...
I20260810 23:01:42.400945 27237 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:01:42.402566 26318 tablet_copy_service.cc:342] P 0f99da2aa72a42b798e738c6a97b4c14: Request end of tablet copy session f7ebe0dbffb74ffdaaa35b4b39c8f037-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.56.60@KRBTEST.COM'} at 127.0.0.1:45606
I20260810 23:01:42.402643 26318 tablet_copy_service.cc:433] P 0f99da2aa72a42b798e738c6a97b4c14: ending tablet copy session f7ebe0dbffb74ffdaaa35b4b39c8f037-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f7ebe0dbffb74ffdaaa35b4b39c8f037
I20260810 23:01:42.410310 27237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:42.411931 27572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:42.412016 27576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:42.412457 27573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:42.412523 27237 server_base.cc:1034] running on GCE node
I20260810 23:01:42.412770 27237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:42.413105 27237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:42.414240 27237 hybrid_clock.cc:648] HybridClock initialized: now 1786402902414239 us; error 33 us; skew 500 ppm
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:42.415650 27237 init.cc:377] Logged in from keytab as oryx/127.15.56.60@KRBTEST.COM (short username oryx)
I20260810 23:01:42.416113 27237 webserver.cc:460] Webserver started at http://127.15.56.60:39843/ using document root <none> and password file <none>
I20260810 23:01:42.416303 27237 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:42.416339 27237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:42.417379 27237 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:01:42.418246 27594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:42.418458 27237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:01:42.418533 27237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/wal
uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037"
format_stamp: "Formatted at 2026-08-10 23:01:42 on dist-test-slave-6bbx"
server_key: "724139b3ce1b2abd98cb1d3b85a20513"
server_key_iv: "194c3d1f09ed4567128d21acbb531e48"
server_key_version: "encryptionkey@0"
I20260810 23:01:42.418622 27237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:42.430115 27237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:42.431957 27237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:42.436467 27237 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.60:38129
I20260810 23:01:42.436981 27237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
I20260810 23:01:42.438542 27689 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:01:42.439532 27689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: Bootstrap starting.
I20260810 23:01:42.442945 27685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.60:38129 every 8 connection(s)
I20260810 23:01:42.449539 27689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:01:42.449849 27689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: Bootstrap complete.
I20260810 23:01:42.450402 27689 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } }
I20260810 23:01:42.450528 27689 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7ebe0dbffb74ffdaaa35b4b39c8f037, State: Initialized, Role: LEARNER
I20260810 23:01:42.450636 27689 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } }
I20260810 23:01:42.450994 27696 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.451128 27689 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:42.451192 27696 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:01:42.454635 27713 catalog_manager.cc:1259] Loaded cluster ID: 2dff39177dc5487bb114ba9d5d63b58a
I20260810 23:01:42.454770 27713 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: loading cluster ID for follower catalog manager: success
I20260810 23:01:42.456581 27713 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: acquiring CA information for follower catalog manager: success
I20260810 23:01:42.457706 27713 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:01:42.489851 26801 heartbeater.cc:503] Master 127.15.56.62:40785 was elected leader, sending a full tablet report...
W20260810 23:01:42.518745 27549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:42.519043 27549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:42.519088 27549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:42.523222 27549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:42.523303 27549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:42.523363 27549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:42.528995 27549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab
--principal=oryx/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:40683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=40019
--tserver_master_addrs=127.15.56.62:40785,127.15.56.61:39989,127.15.56.60:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:42.530125 27549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:42.531195 27549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:42.541404 27719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:42.541399 27723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:42.541504 27718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:42.541610 27549 server_base.cc:1034] running on GCE node
I20260810 23:01:42.542095 27549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:42.542802 27549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:42.543979 27549 hybrid_clock.cc:648] HybridClock initialized: now 1786402902543931 us; error 60 us; skew 500 ppm
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:42.546787 27549 init.cc:377] Logged in from keytab as oryx/127.15.56.1@KRBTEST.COM (short username oryx)
I20260810 23:01:42.547780 27549 webserver.cc:460] Webserver started at http://127.15.56.1:40019/ using document root <none> and password file <none>
I20260810 23:01:42.548363 27549 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:42.548454 27549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:42.552929 27549 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 23:01:42.555210 27731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:42.556734 27549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:01:42.556883 27549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "55a97d9983524ec28f5aef1a5176efb3"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "df59499d5381c7bf6804b73f7501643d"
server_key_iv: "5b86f41c11ca1902ef2ea08f9e06c428"
server_key_version: "encryptionkey@0"
I20260810 23:01:42.557482 27549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:42.588274 27549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:42.591576 27549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:42.591809 27549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:42.592475 27549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:42.593410 27549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:42.593461 27549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:42.593516 27549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:42.593544 27549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:42.603060 27549 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:40683
I20260810 23:01:42.603091 27915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:40683 every 8 connection(s)
I20260810 23:01:42.604243 27549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:42.606825 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 27549
I20260810 23:01:42.607031 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 26592
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.61@KRBTEST.COM
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.60@KRBTEST.COM
I20260810 23:01:42.624287 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.606340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:36075 (local address 127.15.56.61:39989)
0810 23:01:42.606498 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.606503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.607992 (+ 1489us) server_negotiation.cc:407] Connection header received
0810 23:01:42.609203 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.609208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.609265 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.609314 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.611298 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.612345 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.614685 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.615104 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.615200 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.620216 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.620220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.620222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.620251 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.622050 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.622655 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.622659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.622660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.622742 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.623087 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.623090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.623091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.623296 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.623356 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.624013 (+ 657us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.624134 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:01:42.625169 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.606308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:45009 (local address 127.15.56.60:38129)
0810 23:01:42.606589 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.606593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.611451 (+ 4858us) server_negotiation.cc:407] Connection header received
0810 23:01:42.611633 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.611636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.611706 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.611758 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.612322 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.613218 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.616211 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.616691 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.616814 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.621014 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.621020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.621023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.621060 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.622852 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.623461 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.623465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.623467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.623582 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.623988 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.623992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.623994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.624278 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.624380 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.624816 (+ 436us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.624979 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260810 23:01:42.626224 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.606881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:39989 (local address 127.15.56.1:36075)
0810 23:01:42.607795 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.607853 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.608944 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.609785 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.609796 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.610452 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.611112 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.611130 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.612627 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.612632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.614509 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.614523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.615474 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.615481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.615800 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.617057 (+ 1257us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.617083 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.620013 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.622206 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.622212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.622233 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.622555 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.622842 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.622845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.622846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.623006 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.623425 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.623432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.623759 (+ 327us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.624644 (+ 885us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.625034 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":153,"threads_started":1}
I20260810 23:01:42.626240 27926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.606881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:38129 (local address 127.15.56.1:45009)
0810 23:01:42.611375 (+ 4494us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.611388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.611511 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.611908 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.611911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.612100 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.612198 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.612203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.613382 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.613385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.616092 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.616103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.618092 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.618098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.618214 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.618829 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.618846 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.620803 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.623009 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.623013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.623016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.623315 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.623735 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.623739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.623741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.623856 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.624453 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.624456 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.624578 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.625429 (+ 851us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.625603 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4406,"mutex_wait_us":42,"spinlock_wait_cycles":34560,"thread_start_us":153,"threads_started":1}
I20260810 23:01:42.631194 27916 heartbeater.cc:347] Connected to a master server at 127.15.56.60:38129
I20260810 23:01:42.631557 27916 heartbeater.cc:464] Registering TS with master...
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:42.632484 27916 heartbeater.cc:511] Master 127.15.56.60:38129 requested a full tablet report, sending...
I20260810 23:01:42.634750 27626 ts_manager.cc:194] Registered new tserver with Master: 55a97d9983524ec28f5aef1a5176efb3 (127.15.56.1:40683)
I20260810 23:01:42.635993 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.609924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:36639 (local address 127.15.56.62:40785)
0810 23:01:42.616631 (+ 6707us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.616636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.627002 (+ 10366us) server_negotiation.cc:407] Connection header received
0810 23:01:42.627315 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.627320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.627392 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.627460 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.628247 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.629048 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.629960 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.630231 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.630297 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.632979 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.632983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.632985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.633011 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.634389 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.634809 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.634812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.634814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.634889 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.635172 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.635175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.635176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.635372 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.635433 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.635735 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.635855 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6608}
I20260810 23:01:42.636991 27926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.608072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.1:36639)
0810 23:01:42.626901 (+ 18829us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.626920 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.627097 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.627729 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.627732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.627935 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.628075 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.628085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.629174 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.629177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.629848 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.629856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.630445 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.630448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.630530 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.631421 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.631433 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.632839 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.634511 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.634514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.634516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.634711 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.635017 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.635019 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.635021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.635088 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.635481 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.635483 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.635561 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.636608 (+ 1047us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.636793 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18750,"thread_start_us":1524,"threads_started":1}
I20260810 23:01:42.637836 27918 heartbeater.cc:347] Connected to a master server at 127.15.56.62:40785
I20260810 23:01:42.637915 27918 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:42.638080 27918 heartbeater.cc:511] Master 127.15.56.62:40785 requested a full tablet report, sending...
I20260810 23:01:42.638908 26288 ts_manager.cc:194] Re-registered known tserver with Master: 55a97d9983524ec28f5aef1a5176efb3 (127.15.56.1:40683)
I20260810 23:01:42.639555 26288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:36639
I20260810 23:01:42.643940 27919 heartbeater.cc:347] Connected to a master server at 127.15.56.61:39989
I20260810 23:01:42.644068 27919 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:42.644312 27919 heartbeater.cc:511] Master 127.15.56.61:39989 requested a full tablet report, sending...
I20260810 23:01:42.645501 27278 ts_manager.cc:194] Registered new tserver with Master: 55a97d9983524ec28f5aef1a5176efb3 (127.15.56.1:40683)
I20260810 23:01:42.649773 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:34583
--local_ip_for_outbound_sockets=127.15.56.2
--tserver_master_addrs=127.15.56.62:40785
--webserver_port=43827
--webserver_interface=127.15.56.2
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab
--principal=oryx/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.56.62:40785,127.15.56.61:39989
--tserver_master_addrs=127.15.56.62:40785,127.15.56.61:39989,127.15.56.60:38129 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:01:42.721114 27039 heartbeater.cc:503] Master 127.15.56.62:40785 was elected leader, sending a full tablet report...
W20260810 23:01:42.772395 27928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:42.772680 27928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:42.772730 27928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:42.776769 27928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:42.776834 27928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:42.776906 27928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:42.782441 27928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab
--principal=oryx/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:34583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=43827
--tserver_master_addrs=127.15.56.62:40785,127.15.56.61:39989,127.15.56.60:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:42.783694 27928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:42.784834 27928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:42.795251 27943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:42.795248 27942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:42.795254 27947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:42.795696 27928 server_base.cc:1034] running on GCE node
I20260810 23:01:42.796169 27928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:42.796877 27928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:42.798067 27928 hybrid_clock.cc:648] HybridClock initialized: now 1786402902798038 us; error 37 us; skew 500 ppm
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:42.801167 27928 init.cc:377] Logged in from keytab as oryx/127.15.56.2@KRBTEST.COM (short username oryx)
I20260810 23:01:42.802222 27928 webserver.cc:460] Webserver started at http://127.15.56.2:43827/ using document root <none> and password file <none>
I20260810 23:01:42.802814 27928 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:42.802906 27928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:42.807399 27928 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:42.810343 27964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:42.811977 27928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:01:42.812130 27928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "166e4b15b20145908fb57ace1c03ce7f"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "b3b0a57a1d7f149c2fb86745701a0371"
server_key_iv: "1aee984f59573a0cbbe2836b28403a38"
server_key_version: "encryptionkey@0"
I20260810 23:01:42.812801 27928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:42.835427 27928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:42.839311 27928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:42.839538 27928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:42.840317 27928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:42.841360 27928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:42.841423 27928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:42.841482 27928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:42.841511 27928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:42.850443 27928 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:34583
I20260810 23:01:42.850463 28138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:34583 every 8 connection(s)
I20260810 23:01:42.851718 27928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:42.857999 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 27928
I20260810 23:01:42.858165 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 26814
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.60@KRBTEST.COM
I20260810 23:01:42.869407 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.853846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:46549 (local address 127.15.56.60:38129)
0810 23:01:42.853954 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.853958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.854987 (+ 1029us) server_negotiation.cc:407] Connection header received
0810 23:01:42.856224 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.856228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.856296 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.856345 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.857921 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.858741 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.862272 (+ 3531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.862509 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.862567 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.865459 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.865463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.865466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.865496 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.866888 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.867507 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.867511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.867513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.867594 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.867986 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.867988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.867989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.868197 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.868285 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.868994 (+ 709us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.869183 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 23:01:42.871194 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.854266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:38129 (local address 127.15.56.2:46549)
0810 23:01:42.854811 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.854876 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.855982 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.856582 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.856589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.857134 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.857735 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.857755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.861049 (+ 3294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.861052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.862138 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.862148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.862770 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.862773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.862966 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.863593 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.863612 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.865265 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.867036 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.867042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.867060 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.867401 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.867750 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.867752 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.867754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.867898 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.868347 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.868353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.868662 (+ 309us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.869713 (+ 1051us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.870164 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":107,"spinlock_wait_cycles":21760,"thread_start_us":140,"threads_started":1}
I20260810 23:01:42.872530 28140 heartbeater.cc:347] Connected to a master server at 127.15.56.60:38129
I20260810 23:01:42.872771 28140 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:42.873366 28140 heartbeater.cc:511] Master 127.15.56.60:38129 requested a full tablet report, sending...
I20260810 23:01:42.874562 27626 ts_manager.cc:194] Registered new tserver with Master: 166e4b15b20145908fb57ace1c03ce7f (127.15.56.2:34583)
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:42.890344 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.853682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:45243 (local address 127.15.56.62:40785)
0810 23:01:42.853837 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.853842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.854988 (+ 1146us) server_negotiation.cc:407] Connection header received
0810 23:01:42.856224 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.856228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.856289 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.856344 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.857920 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.858741 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.884707 (+ 25966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.884949 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.885052 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.887188 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.887192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.887194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.887222 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.888728 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.889167 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.889170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.889172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.889241 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.889520 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.889522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.889524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.889718 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.889778 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.890063 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.890176 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:01:42.890870 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.854266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.2:45243)
0810 23:01:42.854811 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.854876 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.855981 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.856582 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.856589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.857131 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.857735 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.857755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.883926 (+ 26171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.883929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.884553 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.884562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.885074 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.885079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.885182 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.885726 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.885739 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.887023 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.888855 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.888858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.888860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.889076 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.889341 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.889343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.889345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.889433 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.889835 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.889837 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.889923 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.890533 (+ 610us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.890683 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":121,"spinlock_wait_cycles":99712,"thread_start_us":160,"threads_started":1}
I20260810 23:01:42.892405 28144 heartbeater.cc:347] Connected to a master server at 127.15.56.62:40785
I20260810 23:01:42.892525 28144 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:42.892733 28144 heartbeater.cc:511] Master 127.15.56.62:40785 requested a full tablet report, sending...
I20260810 23:01:42.893819 26288 ts_manager.cc:194] Re-registered known tserver with Master: 166e4b15b20145908fb57ace1c03ce7f (127.15.56.2:34583)
I20260810 23:01:42.894495 26288 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:45243
I20260810 23:01:42.896072 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.879413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:38129 (local address 127.0.0.1:47728)
0810 23:01:42.888655 (+ 9242us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.888678 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.888812 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.892052 (+ 3240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.892056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.892080 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:42.892178 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.892185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.893202 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.893205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.894212 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.894223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.895612 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.895616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.895774 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.895796 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.895871 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.895950 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9114}
I20260810 23:01:42.896576 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.879625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.15.56.60:38129)
0810 23:01:42.879786 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.879790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.891038 (+ 11248us) server_negotiation.cc:407] Connection header received
0810 23:01:42.891126 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.891130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.891228 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.891381 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:42.892357 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.893069 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.894581 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.895201 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.895310 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.895489 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.896375 (+ 886us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.896462 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:01:42.897182 27640 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 23:01:42.897697 27696 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f99da2aa72a42b798e738c6a97b4c14. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:42.897845 27696 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: This master's current role is: LEARNER
Aug 10 23:01:42 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402902, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.61@KRBTEST.COM
I20260810 23:01:42.900377 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:45637
--local_ip_for_outbound_sockets=127.15.56.3
--tserver_master_addrs=127.15.56.62:40785
--webserver_port=38799
--webserver_interface=127.15.56.3
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab
--principal=oryx/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.56.62:40785,127.15.56.61:39989
--tserver_master_addrs=127.15.56.62:40785,127.15.56.61:39989,127.15.56.60:38129 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:01:42.902281 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.855508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:56423 (local address 127.15.56.61:39989)
0810 23:01:42.855643 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:42.855648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:42.855794 (+ 146us) server_negotiation.cc:407] Connection header received
0810 23:01:42.856198 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:42.856202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:42.856260 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:42.856342 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:42.857922 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.858707 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.894435 (+ 35728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.894804 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.894894 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.898872 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:42.898876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:42.898878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:42.898907 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:42.900570 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.901041 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.901045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.901046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.901134 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:42.901410 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:42.901413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:42.901415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:42.901616 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:42.901703 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:42.901988 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 23:01:42.902127 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:01:42.902311 28154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:42.855383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:39989 (local address 127.15.56.2:56423)
0810 23:01:42.855688 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:42.855702 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:42.855979 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:42.856580 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:42.856588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:42.857128 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:42.857735 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.857755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.893349 (+ 35594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.893352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.894276 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:42.894289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:42.895151 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:42.895154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:42.895269 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:42.896021 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:42.896035 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:42.898662 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:42.900698 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.900701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.900704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.900919 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.901224 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:42.901227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:42.901230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:42.901313 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:42.901707 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:42.901711 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:42.901819 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 23:01:42.901901 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:01:42.902083 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":222,"thread_start_us":103,"threads_started":1}
I20260810 23:01:42.903128 28149 heartbeater.cc:347] Connected to a master server at 127.15.56.61:39989
I20260810 23:01:42.903240 28149 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:42.903479 28149 heartbeater.cc:511] Master 127.15.56.61:39989 requested a full tablet report, sending...
I20260810 23:01:42.904254 27278 ts_manager.cc:194] Registered new tserver with Master: 166e4b15b20145908fb57ace1c03ce7f (127.15.56.2:34583)
W20260810 23:01:43.021311 28157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:43.021621 28157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:43.021667 28157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:43.025803 28157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:43.025883 28157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:43.025959 28157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:43.032241 28157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab
--principal=oryx/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:45637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=38799
--tserver_master_addrs=127.15.56.62:40785,127.15.56.61:39989,127.15.56.60:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:43.033478 28157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:43.034611 28157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:43.045143 28169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:43.045138 28172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:43.045138 28168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:43.045415 28157 server_base.cc:1034] running on GCE node
I20260810 23:01:43.045979 28157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:43.046689 28157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:43.047886 28157 hybrid_clock.cc:648] HybridClock initialized: now 1786402903047864 us; error 31 us; skew 500 ppm
Aug 10 23:01:43 dist-test-slave-6bbx krb5kdc[26208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402903, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:43.050702 28157 init.cc:377] Logged in from keytab as oryx/127.15.56.3@KRBTEST.COM (short username oryx)
I20260810 23:01:43.051631 28157 webserver.cc:460] Webserver started at http://127.15.56.3:38799/ using document root <none> and password file <none>
I20260810 23:01:43.052273 28157 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:43.052358 28157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:43.056325 28157 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:01:43.058415 28183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:43.059726 28157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:01:43.059850 28157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "14835bd4d1074a1194177fdf78deacab"
format_stamp: "Formatted at 2026-08-10 23:01:41 on dist-test-slave-6bbx"
server_key: "0ba9a0f5b5290f400779fd3aab6d8e7b"
server_key_iv: "791650d58af80afd78164831d57bdcde"
server_key_version: "encryptionkey@0"
I20260810 23:01:43.060461 28157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:43.086813 28157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:43.091208 28157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:43.091463 28157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:43.092180 28157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:43.093130 28157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:43.093180 28157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:43.093235 28157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:43.093264 28157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:43.103695 28157 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:45637
I20260810 23:01:43.103729 28382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:45637 every 8 connection(s)
I20260810 23:01:43.104894 28157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:43.108392 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 28157
Aug 10 23:01:43 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402903, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.60@KRBTEST.COM
Aug 10 23:01:43 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402903, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:43.123920 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.111068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.15.56.61:39989)
0810 23:01:43.111397 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.111401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.111738 (+ 337us) server_negotiation.cc:407] Connection header received
0810 23:01:43.111963 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.111967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.112021 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.112123 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:43.113418 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.114278 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.115520 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.115918 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.116509 (+ 591us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.118654 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:43.118657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:43.118660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:43.118690 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:43.121958 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.122458 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.122461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.122463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.122558 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.122872 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.122875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.122877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.123085 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:43.123189 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.123565 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.123727 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":93,"threads_started":1}
I20260810 23:01:43.124032 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.108075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:40651 (local address 127.15.56.60:38129)
0810 23:01:43.108207 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.108212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.109637 (+ 1425us) server_negotiation.cc:407] Connection header received
0810 23:01:43.110175 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.110179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.110241 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.110295 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:43.112411 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.113151 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.114902 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.115209 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.115278 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.118865 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:43.118869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:43.118871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:43.118899 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:43.121304 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.122051 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.122055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.122057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.122157 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.122615 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.122619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.122621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.122872 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:43.122944 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.123742 (+ 798us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.123865 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:01:43.124557 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.111657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.15.56.60:38129)
0810 23:01:43.111960 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.111964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.112629 (+ 665us) server_negotiation.cc:407] Connection header received
0810 23:01:43.112817 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.112821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.112875 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.112998 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:43.113907 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.114640 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.115689 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.116019 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.116088 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.117874 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:43.117878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:43.117882 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:43.117912 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:43.119761 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.123181 (+ 3420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.123184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.123186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.123293 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.123580 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.123584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.123585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.123819 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:43.123920 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.124287 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.124406 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260810 23:01:43.125960 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.111402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.15.56.62:40785)
0810 23:01:43.112094 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.112098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.112110 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:01:43.112151 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.112154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.112200 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.112304 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:43.113160 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.114010 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.115176 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.115486 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.116013 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.119873 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:43.119876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:43.119879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:43.119897 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:43.121596 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.124252 (+ 2656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.124256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.124258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.124350 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.124693 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.124697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.124699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.124912 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:43.125564 (+ 652us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.125662 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.125792 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":81,"threads_started":1}
Aug 10 23:01:43 dist-test-slave-6bbx krb5kdc[26208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402903, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.61@KRBTEST.COM
I20260810 23:01:43.128197 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.107977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:38129 (local address 127.15.56.3:40651)
0810 23:01:43.109535 (+ 1558us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:43.109549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:43.109915 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:43.110767 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:43.110778 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:43.111370 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:43.112230 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.112251 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.113322 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.113326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.114764 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.114776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.115491 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.115495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.115750 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.116643 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:43.116668 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:43.118666 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:43.121445 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:43.121453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:43.121478 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:43.121940 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:43.122279 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:43.122282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:43.122284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:43.122501 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:43.123035 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:43.123042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:43.123404 (+ 362us) client_negotiation.cc:769] Sending connection context
0810 23:01:43.126145 (+ 2741us) client_negotiation.cc:240] Negotiation successful
0810 23:01:43.126584 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1491,"thread_start_us":156,"threads_started":1}
I20260810 23:01:43.129478 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.107497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.3:58839)
0810 23:01:43.108656 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:43.108722 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:43.109907 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:43.110767 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:43.110777 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:43.118573 (+ 7796us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:43.118699 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.118704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.119602 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.119604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.120472 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.120486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.121191 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.121194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.121300 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.121862 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:43.121874 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:43.124861 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:43.127433 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:43.127436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:43.127438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:43.127706 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:43.128153 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:43.128156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:43.128159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:43.128257 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:43.128779 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:43.128782 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:43.128898 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 23:01:43.129002 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 23:01:43.129166 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":753,"threads_started":1}
I20260810 23:01:43.130245 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.107064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:58839 (local address 127.15.56.62:40785)
0810 23:01:43.107202 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.107207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.108846 (+ 1639us) server_negotiation.cc:407] Connection header received
0810 23:01:43.110189 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.110193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.110257 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.110333 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:43.118846 (+ 8513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.119455 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.120621 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.120933 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.121013 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.125009 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:43.125013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:43.125015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:43.125037 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:43.127067 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.127833 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.127836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.127838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.127927 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.128391 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.128394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.128396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.128629 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:43.128723 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.130013 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.130132 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:01:43.132823 28383 heartbeater.cc:347] Connected to a master server at 127.15.56.60:38129
I20260810 23:01:43.133068 28384 heartbeater.cc:347] Connected to a master server at 127.15.56.62:40785
I20260810 23:01:43.133131 28383 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:43.133186 28384 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:43.133484 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.109214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:43173 (local address 127.15.56.61:39989)
0810 23:01:43.109336 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.109340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.110002 (+ 662us) server_negotiation.cc:407] Connection header received
0810 23:01:43.110187 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.110190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.110241 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.110295 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:43.117444 (+ 7149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.118235 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.123160 (+ 4925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.123465 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.123563 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.128683 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:43.128687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:43.128690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:43.128724 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:43.131033 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.131637 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.131642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.131643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.131789 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.132427 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.132430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.132432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.132682 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:43.132782 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.133181 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.133329 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:01:43.133803 28383 heartbeater.cc:511] Master 127.15.56.60:38129 requested a full tablet report, sending...
I20260810 23:01:43.134065 28387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.109053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:39989 (local address 127.15.56.3:43173)
0810 23:01:43.109915 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:43.109929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:43.110067 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:43.110767 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:43.110778 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:43.116265 (+ 5487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:43.116422 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.116431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.118419 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.118816 (+ 397us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.123025 (+ 4209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.123035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.125376 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.125381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.125473 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.125880 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:43.125894 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:43.128247 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:43.131291 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:43.131294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:43.131295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:43.131511 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:43.131922 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:43.131925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:43.131927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:43.132049 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:43.132836 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:43.132840 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:43.132958 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 23:01:43.133699 (+ 741us) client_negotiation.cc:240] Negotiation successful
0810 23:01:43.133873 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":86,"threads_started":1}
I20260810 23:01:43.134464 28384 heartbeater.cc:511] Master 127.15.56.62:40785 requested a full tablet report, sending...
I20260810 23:01:43.130424 26288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:01:43.134886 28385 heartbeater.cc:347] Connected to a master server at 127.15.56.61:39989
I20260810 23:01:43.134974 28385 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:43.135172 28385 heartbeater.cc:511] Master 127.15.56.61:39989 requested a full tablet report, sending...
I20260810 23:01:43.135288 26287 ts_manager.cc:194] Re-registered known tserver with Master: 14835bd4d1074a1194177fdf78deacab (127.15.56.3:45637)
I20260810 23:01:43.135756 27626 ts_manager.cc:194] Registered new tserver with Master: 14835bd4d1074a1194177fdf78deacab (127.15.56.3:45637)
W20260810 23:01:43.136068 26288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:43.136065 26287 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:58839
I20260810 23:01:43.136415 27278 ts_manager.cc:194] Registered new tserver with Master: 14835bd4d1074a1194177fdf78deacab (127.15.56.3:45637)
I20260810 23:01:43.138989 27560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: NON_VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: true } } }
I20260810 23:01:43.139163 27560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:43.143252 28412 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14: attempting to promote NON_VOTER f7ebe0dbffb74ffdaaa35b4b39c8f037 to VOTER
I20260810 23:01:43.143656 28412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } }
I20260810 23:01:43.145576 27640 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 LEARNER]: Refusing update from remote peer 0f99da2aa72a42b798e738c6a97b4c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 23:01:43.146097 28412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 23:01:43.147812 27298 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Refusing update from remote peer 0f99da2aa72a42b798e738c6a97b4c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 23:01:43.148422 27560 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f7ebe0dbffb74ffdaaa35b4b39c8f037 (127.15.56.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:43.148568 28411 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 23:01:43.149073 27640 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f7ebe0dbffb74ffdaaa35b4b39c8f037 (127.15.56.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:43.149802 27560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:43.149915 27560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:43.149879 28412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:43.149981 28412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:43.151149 27696 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:43.151279 27696 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:43.152295 27298 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f7ebe0dbffb74ffdaaa35b4b39c8f037 (127.15.56.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:43.153882 28426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:43.154035 28426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:43.156914 28409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f99da2aa72a42b798e738c6a97b4c14" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:43.157070 28409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:43.161912 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.155922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:40683 (local address 127.0.0.1:39062)
0810 23:01:43.156097 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:43.156114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:43.156218 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:43.157654 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:43.157658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:43.157676 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:43.157801 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.157808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.159620 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.159623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.160525 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.160537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.161545 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.161548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.161613 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.161639 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:01:43.161702 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:01:43.161791 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":88}
I20260810 23:01:43.162127 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.156130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.15.56.1:40683)
0810 23:01:43.157086 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.157103 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.157129 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:01:43.157217 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.157224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.157495 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.158527 (+ 1032us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:43.158680 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.159497 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.160652 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.161424 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.161540 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.161742 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.161858 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.161974 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":129,"threads_started":1}
I20260810 23:01:43.163592 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.156722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:34583 (local address 127.0.0.1:47314)
0810 23:01:43.157103 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:43.157117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:43.157202 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:43.158005 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:43.158008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:43.158028 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:43.158361 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.158367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.161001 (+ 2634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.161004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.162088 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.162114 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.163240 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.163244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.163341 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.163366 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:01:43.163419 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 23:01:43.163487 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":239,"thread_start_us":104,"threads_started":1}
I20260810 23:01:43.164180 28432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.157170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.15.56.2:34583)
0810 23:01:43.157496 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.157508 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.157531 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:01:43.157610 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.157616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.157841 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.158101 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:43.159646 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.160880 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.162257 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.163087 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.163787 (+ 700us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.163930 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.163983 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.164068 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":121,"threads_started":1}
I20260810 23:01:43.164426 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.156218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:45637 (local address 127.0.0.1:36302)
0810 23:01:43.156721 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:43.156734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:43.156841 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:43.157893 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:43.157896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:43.157912 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:43.158180 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.158190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.159610 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.159613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.160982 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.160992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.163977 (+ 2985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.163981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.164125 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.164150 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:01:43.164226 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:01:43.164302 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":29,"thread_start_us":132,"threads_started":1}
I20260810 23:01:43.164832 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.156675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.15.56.3:45637)
0810 23:01:43.157452 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.157465 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.157487 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:01:43.157564 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.157569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.157774 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.157935 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:43.158754 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.159489 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.162315 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.163112 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.163214 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.163379 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.164432 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.164679 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":138,"threads_started":1}
I20260810 23:01:43.165501 27811 tablet_service.cc:1467] Processing CreateTablet for tablet 919ed68d97624ada8391a30743cc8123 (DEFAULT_TABLE table=test-table [id=7a27508a28754390aa8161d7f00be399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:43.166508 27811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919ed68d97624ada8391a30743cc8123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:43.166778 28041 tablet_service.cc:1467] Processing CreateTablet for tablet 919ed68d97624ada8391a30743cc8123 (DEFAULT_TABLE table=test-table [id=7a27508a28754390aa8161d7f00be399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:43.167817 28041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919ed68d97624ada8391a30743cc8123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:43.173599 28434 tablet_bootstrap.cc:492] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: Bootstrap starting.
I20260810 23:01:43.175002 28437 tablet_bootstrap.cc:492] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: Bootstrap starting.
I20260810 23:01:43.177129 28437 tablet_bootstrap.cc:654] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:43.177253 28434 tablet_bootstrap.cc:654] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:43.177945 28437 log.cc:824] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:43.177489 28275 tablet_service.cc:1467] Processing CreateTablet for tablet 919ed68d97624ada8391a30743cc8123 (DEFAULT_TABLE table=test-table [id=7a27508a28754390aa8161d7f00be399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:43.178126 28434 log.cc:824] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:43.178730 28275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919ed68d97624ada8391a30743cc8123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:43.180469 28437 tablet_bootstrap.cc:492] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: No bootstrap required, opened a new log
I20260810 23:01:43.180691 28437 ts_tablet_manager.cc:1397] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:01:43.181382 28434 tablet_bootstrap.cc:492] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: No bootstrap required, opened a new log
I20260810 23:01:43.181591 28434 ts_tablet_manager.cc:1397] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 23:01:43.184872 28434 raft_consensus.cc:348] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:43.185117 28434 raft_consensus.cc:374] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:43.185173 28434 raft_consensus.cc:729] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a97d9983524ec28f5aef1a5176efb3, State: Initialized, Role: FOLLOWER
I20260810 23:01:43.185339 28437 raft_consensus.cc:348] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:43.185608 28437 raft_consensus.cc:374] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:43.185674 28437 raft_consensus.cc:729] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166e4b15b20145908fb57ace1c03ce7f, State: Initialized, Role: FOLLOWER
I20260810 23:01:43.185611 28434 consensus_queue.cc:260] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:43.186115 28437 consensus_queue.cc:260] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:43.186614 28434 ts_tablet_manager.cc:1428] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260810 23:01:43.186919 28144 heartbeater.cc:503] Master 127.15.56.62:40785 was elected leader, sending a full tablet report...
I20260810 23:01:43.187152 28437 ts_tablet_manager.cc:1428] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:01:43.188108 27918 heartbeater.cc:503] Master 127.15.56.62:40785 was elected leader, sending a full tablet report...
I20260810 23:01:43.191883 28449 tablet_bootstrap.cc:492] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: Bootstrap starting.
I20260810 23:01:43.195171 28449 tablet_bootstrap.cc:654] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:43.196041 28449 log.cc:824] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:43.198259 28449 tablet_bootstrap.cc:492] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: No bootstrap required, opened a new log
I20260810 23:01:43.198459 28449 ts_tablet_manager.cc:1397] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 23:01:43.202289 28449 raft_consensus.cc:348] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:43.202576 28449 raft_consensus.cc:374] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:43.202659 28449 raft_consensus.cc:729] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14835bd4d1074a1194177fdf78deacab, State: Initialized, Role: FOLLOWER
I20260810 23:01:43.203131 28449 consensus_queue.cc:260] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:43.203975 28384 heartbeater.cc:503] Master 127.15.56.62:40785 was elected leader, sending a full tablet report...
I20260810 23:01:43.204350 28449 ts_tablet_manager.cc:1428] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260810 23:01:43.353909 28150 tablet.cc:2369] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:01:43.356775 27921 tablet.cc:2369] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:01:43.357175 28386 tablet.cc:2369] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:01:43.370999 28447 raft_consensus.cc:482] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:43.371266 28447 raft_consensus.cc:504] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:43.372558 28447 leader_election.cc:288] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55a97d9983524ec28f5aef1a5176efb3 (127.15.56.1:40683), 14835bd4d1074a1194177fdf78deacab (127.15.56.3:45637)
I20260810 23:01:43.377825 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.372932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:36411 (local address 127.15.56.3:45637)
0810 23:01:43.373062 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.373067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.373177 (+ 110us) server_negotiation.cc:407] Connection header received
0810 23:01:43.373255 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.373258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.373307 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.373380 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:43.373797 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.374604 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.376453 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.377077 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.377192 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.377267 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.377595 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.377677 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:01:43.377825 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.373021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:34501 (local address 127.15.56.1:40683)
0810 23:01:43.373131 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.373135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.373150 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:01:43.373214 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.373218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.373274 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.373372 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:43.373797 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.374590 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.376453 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.377077 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.377192 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.377273 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.377595 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.377677 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:01:43.377876 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.372835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:45637 (local address 127.15.56.2:36411)
0810 23:01:43.373099 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:43.373111 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:43.373185 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:43.373397 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:43.373400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:43.373430 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:43.373676 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.373684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.374725 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.374728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.376254 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.376266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.377206 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.377210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.377422 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.377496 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 23:01:43.377572 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:01:43.377689 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":434,"spinlock_wait_cycles":8320}
I20260810 23:01:43.377935 28154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.372835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:40683 (local address 127.15.56.2:34501)
0810 23:01:43.372948 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:43.372962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:43.373061 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:43.373397 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:43.373400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:43.373428 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:43.373683 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.373690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.374725 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.374728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.376254 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:43.376266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.377206 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.377210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:43.377422 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.377492 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 23:01:43.377572 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:01:43.377689 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":223,"spinlock_wait_cycles":8192}
I20260810 23:01:43.378672 27848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919ed68d97624ada8391a30743cc8123" candidate_uuid: "166e4b15b20145908fb57ace1c03ce7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a97d9983524ec28f5aef1a5176efb3" is_pre_election: true
I20260810 23:01:43.378849 28307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919ed68d97624ada8391a30743cc8123" candidate_uuid: "166e4b15b20145908fb57ace1c03ce7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14835bd4d1074a1194177fdf78deacab" is_pre_election: true
I20260810 23:01:43.379031 27848 raft_consensus.cc:2454] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166e4b15b20145908fb57ace1c03ce7f in term 0.
I20260810 23:01:43.379182 28307 raft_consensus.cc:2454] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166e4b15b20145908fb57ace1c03ce7f in term 0.
I20260810 23:01:43.379586 27965 leader_election.cc:302] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166e4b15b20145908fb57ace1c03ce7f, 55a97d9983524ec28f5aef1a5176efb3; no voters:
I20260810 23:01:43.379917 28447 raft_consensus.cc:2781] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:43.380003 28447 raft_consensus.cc:482] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:43.380038 28447 raft_consensus.cc:3037] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:43.381125 28447 raft_consensus.cc:504] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:43.381488 28447 leader_election.cc:288] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [CANDIDATE]: Term 1 election: Requested vote from peers 55a97d9983524ec28f5aef1a5176efb3 (127.15.56.1:40683), 14835bd4d1074a1194177fdf78deacab (127.15.56.3:45637)
I20260810 23:01:43.381892 28307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919ed68d97624ada8391a30743cc8123" candidate_uuid: "166e4b15b20145908fb57ace1c03ce7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14835bd4d1074a1194177fdf78deacab"
I20260810 23:01:43.382025 28307 raft_consensus.cc:3037] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:43.383127 28307 raft_consensus.cc:2454] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166e4b15b20145908fb57ace1c03ce7f in term 1.
I20260810 23:01:43.383441 27848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919ed68d97624ada8391a30743cc8123" candidate_uuid: "166e4b15b20145908fb57ace1c03ce7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a97d9983524ec28f5aef1a5176efb3"
I20260810 23:01:43.383483 27972 leader_election.cc:302] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14835bd4d1074a1194177fdf78deacab, 166e4b15b20145908fb57ace1c03ce7f; no voters:
I20260810 23:01:43.383579 27848 raft_consensus.cc:3037] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:43.383692 28447 raft_consensus.cc:2781] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:43.383931 28447 raft_consensus.cc:686] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 1 LEADER]: Becoming Leader. State: Replica: 166e4b15b20145908fb57ace1c03ce7f, State: Running, Role: LEADER
I20260810 23:01:43.384358 28447 consensus_queue.cc:237] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:43.385114 27848 raft_consensus.cc:2454] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166e4b15b20145908fb57ace1c03ce7f in term 1.
I20260810 23:01:43.388413 26287 catalog_manager.cc:5495] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 166e4b15b20145908fb57ace1c03ce7f (127.15.56.2). New cstate: current_term: 1 leader_uuid: "166e4b15b20145908fb57ace1c03ce7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } health_report { overall_health: HEALTHY } } }
I20260810 23:01:43.456542 28432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.452780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47322 (local address 127.15.56.2:34583)
0810 23:01:43.452917 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.452920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.452935 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:01:43.453039 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.453042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.453103 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.453214 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:43.453495 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.454048 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.454922 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.455159 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.455244 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.455524 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:43.456029 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:43.456126 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.456349 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.456419 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:01:43.463236 28307 raft_consensus.cc:1261] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab [term 1 FOLLOWER]: Refusing update from remote peer 166e4b15b20145908fb57ace1c03ce7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:43.463236 27848 raft_consensus.cc:1261] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3 [term 1 FOLLOWER]: Refusing update from remote peer 166e4b15b20145908fb57ace1c03ce7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:43.464102 28455 consensus_queue.cc:1035] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:43.464298 28447 consensus_queue.cc:1035] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:43.475706 28465 mvcc.cc:205] Tried to move back new op lower bound from 7317106292576735232 to 7317106292266020864. Current Snapshot: MvccSnapshot[applied={T|T < 7317106292576735232 or (T in {7317106292576735232})}]
I20260810 23:01:43.487358 26287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:43.487588 26287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:43.492137 26287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f99da2aa72a42b798e738c6a97b4c14: Sending DeleteTablet for 3 replicas of tablet 919ed68d97624ada8391a30743cc8123
I20260810 23:01:43.493237 27811 tablet_service.cc:1514] Processing DeleteTablet for tablet 919ed68d97624ada8391a30743cc8123 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:43 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:39062
I20260810 23:01:43.493237 28275 tablet_service.cc:1514] Processing DeleteTablet for tablet 919ed68d97624ada8391a30743cc8123 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:43 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:36302
I20260810 23:01:43.494225 28041 tablet_service.cc:1514] Processing DeleteTablet for tablet 919ed68d97624ada8391a30743cc8123 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:43 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:47314
I20260810 23:01:43.494712 28481 tablet_replica.cc:330] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: stopping tablet replica
I20260810 23:01:43.495159 28481 raft_consensus.cc:2229] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:43.495641 28481 raft_consensus.cc:2258] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:43.495795 28482 tablet_replica.cc:330] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: stopping tablet replica
I20260810 23:01:43.496189 28482 raft_consensus.cc:2229] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:43.496330 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 26228
I20260810 23:01:43.502830 28482 raft_consensus.cc:2258] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:43.503641 28480 tablet_replica.cc:330] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: stopping tablet replica
I20260810 23:01:43.504074 28480 raft_consensus.cc:2229] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:43.504405 28480 raft_consensus.cc:2258] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:43.505472 28481 ts_tablet_manager.cc:1905] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:43.507277 28482 ts_tablet_manager.cc:1905] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:43.508658 28481 ts_tablet_manager.cc:1918] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:43.508774 28481 log.cc:1196] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/919ed68d97624ada8391a30743cc8123
I20260810 23:01:43.509121 28481 ts_tablet_manager.cc:1939] T 919ed68d97624ada8391a30743cc8123 P 14835bd4d1074a1194177fdf78deacab: Deleting consensus metadata
I20260810 23:01:43.510191 28482 ts_tablet_manager.cc:1918] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:43.510294 28482 log.cc:1196] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/919ed68d97624ada8391a30743cc8123
I20260810 23:01:43.510583 28482 ts_tablet_manager.cc:1939] T 919ed68d97624ada8391a30743cc8123 P 166e4b15b20145908fb57ace1c03ce7f: Deleting consensus metadata
I20260810 23:01:43.514905 28480 ts_tablet_manager.cc:1905] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:01:43.515739 27966 connection.cc:535] server connection from 127.0.0.1:47314 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 23:01:43.516283 28188 connection.cc:535] server connection from 127.0.0.1:36302 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:01:43.517789 28480 ts_tablet_manager.cc:1918] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:43.517901 28480 log.cc:1196] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-0/wal/wals/919ed68d97624ada8391a30743cc8123
I20260810 23:01:43.518234 28480 ts_tablet_manager.cc:1939] T 919ed68d97624ada8391a30743cc8123 P 55a97d9983524ec28f5aef1a5176efb3: Deleting consensus metadata
W20260810 23:01:43.519819 27736 connection.cc:441] server connection from 127.0.0.1:39062 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39062 (request call id 1) could send its response
I20260810 23:01:43.520419 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.495124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.15.56.61:39989)
0810 23:01:43.495656 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.495691 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.503700 (+ 8009us) server_negotiation.cc:407] Connection header received
0810 23:01:43.503990 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.503995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.504062 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.504126 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:43.511280 (+ 7154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.512203 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.513490 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.513878 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.513979 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.516435 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:43.516439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:43.516442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:43.516474 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:43.518250 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.518824 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.518828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.518830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.518925 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.519237 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.519240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.519242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.519554 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:43.519717 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.520126 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.520255 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438}
I20260810 23:01:43.530516 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.495641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.15.56.60:38129)
0810 23:01:43.495794 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.495799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.520971 (+ 25172us) server_negotiation.cc:407] Connection header received
0810 23:01:43.521163 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.521167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.521230 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.521300 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:43.522169 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.522942 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.524118 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.524470 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.524541 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.526496 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:43.526500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:43.526503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:43.526530 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:43.528629 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.529089 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.529093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.529095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.529179 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.529494 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.529497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.529499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.529727 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:43.529831 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.530181 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.530340 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:01:43.530541 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.521589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.15.56.61:39989)
0810 23:01:43.521697 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.521702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.522468 (+ 766us) server_negotiation.cc:407] Connection header received
0810 23:01:43.522617 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.522620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.522660 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.522702 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:43.523517 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.524244 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.525332 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.525651 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.525739 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.527337 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:43.527341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:43.527344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:43.527373 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:43.528875 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.529304 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.529307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.529309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.529386 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.529675 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.529678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.529680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.529905 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:43.529993 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.530301 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.530412 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:01:43.531944 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:43.521936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.15.56.60:38129)
0810 23:01:43.522044 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:43.522048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:43.522561 (+ 513us) server_negotiation.cc:407] Connection header received
0810 23:01:43.522729 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:43.522733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:43.522778 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:43.522827 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:43.523740 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.524512 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.525718 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:43.526066 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:43.526165 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:43.527724 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:43.527728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:43.527730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:43.527756 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:43.529642 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.530094 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.530098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.530099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.530210 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:43.530492 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:43.530496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:43.530497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:43.530694 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:43.530813 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:43.531620 (+ 807us) server_negotiation.cc:299] Negotiation successful
0810 23:01:43.531793 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:01:44.478945 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:44.478253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.2:33371)
0810 23:01:44.478517 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:44.478626 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
W20260810 23:01:44.479422 28144 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:40785 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
I20260810 23:01:44.479831 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:44.479016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.1:51015)
0810 23:01:44.479603 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:44.479729 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":77,"threads_started":1}
W20260810 23:01:44.480518 27918 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:40785 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
I20260810 23:01:44.480587 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:44.480086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.3:48217)
0810 23:01:44.480403 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:44.480490 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
W20260810 23:01:44.480815 28384 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:40785 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
I20260810 23:01:44.993220 28500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0f99da2aa72a42b798e738c6a97b4c14)
I20260810 23:01:44.993276 28501 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0f99da2aa72a42b798e738c6a97b4c14)
I20260810 23:01:44.993448 28500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } }
I20260810 23:01:44.993476 28501 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } }
I20260810 23:01:44.994486 28501 leader_election.cc:288] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785), 78fe7d9db0874fccbd42624a4e89c37f (127.15.56.61:39989)
I20260810 23:01:44.994496 28500 leader_election.cc:288] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785), f7ebe0dbffb74ffdaaa35b4b39c8f037 (127.15.56.60:38129)
I20260810 23:01:44.995131 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:44.994680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46100)
0810 23:01:44.994913 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:44.995002 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":76,"threads_started":1}
I20260810 23:01:44.995208 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:44.994746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46108)
0810 23:01:44.995032 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:44.995107 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":64,"threads_started":1}
W20260810 23:01:44.995460 27600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
W20260810 23:01:44.995534 27252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
I20260810 23:01:44.996412 28505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:44.996145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46110)
0810 23:01:44.996288 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:44.996342 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260810 23:01:44.996560 27252 leader_election.cc:334] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785): Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
I20260810 23:01:44.996719 28506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:44.996382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46126)
0810 23:01:44.996582 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:44.996651 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260810 23:01:44.997215 27600 leader_election.cc:334] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785): Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
I20260810 23:01:44.999323 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:44.994757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:38129 (local address 127.0.0.1:47778)
0810 23:01:44.995383 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:44.995398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:44.995506 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:44.995969 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:44.995972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:44.995993 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:44.996101 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:44.996108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:44.997129 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:44.997132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:44.998269 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:44.998280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:44.998925 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:44.998928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:44.999017 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:44.999041 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:01:44.999117 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:01:44.999196 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":77,"threads_started":1}
I20260810 23:01:44.999325 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:44.995143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47778 (local address 127.15.56.60:38129)
0810 23:01:44.995451 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:44.995455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:44.995469 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:01:44.995653 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:44.995656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:44.995752 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:44.995835 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:44.996222 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:44.996978 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:44.998384 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:44.998813 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:44.998909 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:44.998972 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:44.999140 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 23:01:44.999213 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":241,"thread_start_us":77,"threads_started":1}
I20260810 23:01:44.999856 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78fe7d9db0874fccbd42624a4e89c37f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" is_pre_election: true
I20260810 23:01:44.999966 28509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:44.995308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.15.56.61:39989)
0810 23:01:44.996270 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:44.996274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:44.996287 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:01:44.996325 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:44.996328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:44.996367 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:44.996441 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:44.996932 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:44.997617 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:44.998835 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:44.999229 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:44.999327 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:44.999384 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:44.999763 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 23:01:44.999844 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":84,"threads_started":1}
I20260810 23:01:44.999990 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:44.995020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:39989 (local address 127.0.0.1:33588)
0810 23:01:44.995940 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:44.995953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:44.996056 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:44.996624 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:44.996627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:44.996696 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:44.996798 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:44.996806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:44.997732 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:44.997735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:44.998713 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:44.998723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:44.999347 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:44.999351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:44.999437 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:44.999458 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:01:44.999735 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 23:01:44.999817 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260810 23:01:45.000124 27640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78fe7d9db0874fccbd42624a4e89c37f in term 1.
I20260810 23:01:45.000469 27298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "78fe7d9db0874fccbd42624a4e89c37f" is_pre_election: true
I20260810 23:01:45.000474 27256 leader_election.cc:302] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78fe7d9db0874fccbd42624a4e89c37f, f7ebe0dbffb74ffdaaa35b4b39c8f037; no voters: 0f99da2aa72a42b798e738c6a97b4c14
I20260810 23:01:45.000674 27298 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ebe0dbffb74ffdaaa35b4b39c8f037 in term 1.
I20260810 23:01:45.000751 28500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:01:45.000819 28500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0f99da2aa72a42b798e738c6a97b4c14)
I20260810 23:01:45.000866 28500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:01:45.001044 27600 leader_election.cc:302] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78fe7d9db0874fccbd42624a4e89c37f, f7ebe0dbffb74ffdaaa35b4b39c8f037; no voters: 0f99da2aa72a42b798e738c6a97b4c14
I20260810 23:01:45.001291 28501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:01:45.001350 28501 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0f99da2aa72a42b798e738c6a97b4c14)
I20260810 23:01:45.001394 28501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:01:45.002076 28500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } }
I20260810 23:01:45.002453 28500 leader_election.cc:288] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 2 election: Requested vote from peers 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785), f7ebe0dbffb74ffdaaa35b4b39c8f037 (127.15.56.60:38129)
I20260810 23:01:45.002422 28501 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } }
I20260810 23:01:45.002797 28501 leader_election.cc:288] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [CANDIDATE]: Term 2 election: Requested vote from peers 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785), 78fe7d9db0874fccbd42624a4e89c37f (127.15.56.61:39989)
I20260810 23:01:45.002810 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78fe7d9db0874fccbd42624a4e89c37f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037"
I20260810 23:01:45.003026 27640 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 78fe7d9db0874fccbd42624a4e89c37f in current term 2: Already voted for candidate f7ebe0dbffb74ffdaaa35b4b39c8f037 in this term.
I20260810 23:01:45.003204 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:45.002978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46142)
0810 23:01:45.003074 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:45.003132 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 23:01:45.003333 27298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "78fe7d9db0874fccbd42624a4e89c37f"
I20260810 23:01:45.003448 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:45.002605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46138)
0810 23:01:45.003179 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:45.003239 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 23:01:45.003482 27298 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f7ebe0dbffb74ffdaaa35b4b39c8f037 in current term 2: Already voted for candidate 78fe7d9db0874fccbd42624a4e89c37f in this term.
I20260810 23:01:45.003891 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:45.003648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46150)
0810 23:01:45.003773 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:45.003824 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 23:01:45.003981 27600 leader_election.cc:334] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785): Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
I20260810 23:01:45.004052 27600 leader_election.cc:302] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7ebe0dbffb74ffdaaa35b4b39c8f037; no voters: 0f99da2aa72a42b798e738c6a97b4c14, 78fe7d9db0874fccbd42624a4e89c37f
I20260810 23:01:45.004179 28501 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 23:01:45.004294 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:45.004095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46158)
0810 23:01:45.004196 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:45.004243 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 23:01:45.004393 27252 leader_election.cc:334] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785): Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
I20260810 23:01:45.004457 27252 leader_election.cc:302] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78fe7d9db0874fccbd42624a4e89c37f; no voters: 0f99da2aa72a42b798e738c6a97b4c14, f7ebe0dbffb74ffdaaa35b4b39c8f037
I20260810 23:01:45.004585 28500 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 23:01:45.481186 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:45.480124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.2:37137)
0810 23:01:45.480806 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:45.480888 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":115,"threads_started":1}
I20260810 23:01:45.481793 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:45.481209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.1:57945)
0810 23:01:45.481627 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:45.481695 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":98,"threads_started":1}
I20260810 23:01:45.481839 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:45.481318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.3:42155)
0810 23:01:45.481708 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:45.481758 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":78,"threads_started":1}
I20260810 23:01:46.482981 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.482141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.2:53107)
0810 23:01:46.482758 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.482848 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":88,"threads_started":1}
I20260810 23:01:46.483225 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.482610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.1:45265)
0810 23:01:46.483060 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.483124 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":60,"threads_started":1}
I20260810 23:01:46.483613 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.482603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.3:35731)
0810 23:01:46.483481 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.483538 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":70,"threads_started":1}
I20260810 23:01:46.540992 28523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:46.541143 28523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } }
I20260810 23:01:46.541582 28523 leader_election.cc:288] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785), f7ebe0dbffb74ffdaaa35b4b39c8f037 (127.15.56.60:38129)
I20260810 23:01:46.542096 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78fe7d9db0874fccbd42624a4e89c37f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" is_pre_election: true
I20260810 23:01:46.542225 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.541818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46390)
0810 23:01:46.542049 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.542124 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260810 23:01:46.542284 27640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78fe7d9db0874fccbd42624a4e89c37f in term 2.
I20260810 23:01:46.542654 27256 leader_election.cc:302] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78fe7d9db0874fccbd42624a4e89c37f, f7ebe0dbffb74ffdaaa35b4b39c8f037; no voters:
I20260810 23:01:46.542838 28523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 23:01:46.542899 28523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:46.542938 28523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:01:46.543162 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.542919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46392)
0810 23:01:46.543060 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.543103 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260810 23:01:46.543277 27252 leader_election.cc:334] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785): Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
I20260810 23:01:46.544152 28523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } }
I20260810 23:01:46.544486 28523 leader_election.cc:288] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 3 election: Requested vote from peers 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785), f7ebe0dbffb74ffdaaa35b4b39c8f037 (127.15.56.60:38129)
I20260810 23:01:46.544839 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.544615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46394)
0810 23:01:46.544720 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.544775 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 23:01:46.544865 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "78fe7d9db0874fccbd42624a4e89c37f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037"
I20260810 23:01:46.544988 27640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:01:46.545492 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.545283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46408)
0810 23:01:46.545387 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.545435 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:01:46.545603 27252 leader_election.cc:334] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785): Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
I20260810 23:01:46.546144 27640 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78fe7d9db0874fccbd42624a4e89c37f in term 3.
I20260810 23:01:46.546412 27256 leader_election.cc:302] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78fe7d9db0874fccbd42624a4e89c37f, f7ebe0dbffb74ffdaaa35b4b39c8f037; no voters: 0f99da2aa72a42b798e738c6a97b4c14
I20260810 23:01:46.546561 28523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:01:46.546748 28523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [term 3 LEADER]: Becoming Leader. State: Replica: 78fe7d9db0874fccbd42624a4e89c37f, State: Running, Role: LEADER
I20260810 23:01:46.547091 28523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } }
I20260810 23:01:46.548307 28527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78fe7d9db0874fccbd42624a4e89c37f. Latest consensus state: current_term: 3 leader_uuid: "78fe7d9db0874fccbd42624a4e89c37f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:46.548444 28527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:46.548760 28529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:46.551098 28529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:46.551604 28529 catalog_manager.cc:1259] Loaded cluster ID: 2dff39177dc5487bb114ba9d5d63b58a
I20260810 23:01:46.551684 28529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:46.552311 28529 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:46.552706 28529 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: Loaded TSK: 0
I20260810 23:01:46.553419 28529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:01:46.602988 27278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:01:46.606323 27278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:46.609859 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.609584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46434)
0810 23:01:46.609703 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.609770 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 23:01:46.610031 27640 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [term 3 FOLLOWER]: Refusing update from remote peer 78fe7d9db0874fccbd42624a4e89c37f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 23:01:46.610600 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.610369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.0.0.1:46446)
0810 23:01:46.610458 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.610506 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 23:01:46.610513 28527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 23:01:46.610723 27252 consensus_peers.cc:487] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f -> Peer 0f99da2aa72a42b798e738c6a97b4c14 (127.15.56.62:40785): Couldn't send request to peer 0f99da2aa72a42b798e738c6a97b4c14. Status: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:01:46.612108 28532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78fe7d9db0874fccbd42624a4e89c37f. Latest consensus state: current_term: 3 leader_uuid: "78fe7d9db0874fccbd42624a4e89c37f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:46.612270 28532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:46.613519 28527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "78fe7d9db0874fccbd42624a4e89c37f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:46.613660 28527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:46.614152 28532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "78fe7d9db0874fccbd42624a4e89c37f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f99da2aa72a42b798e738c6a97b4c14" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40785 } } peers { permanent_uuid: "78fe7d9db0874fccbd42624a4e89c37f" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 39989 } attrs { promote: false } } peers { permanent_uuid: "f7ebe0dbffb74ffdaaa35b4b39c8f037" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 38129 } attrs { promote: false } } }
I20260810 23:01:46.614355 28532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ebe0dbffb74ffdaaa35b4b39c8f037 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:46.625522 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.621011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:34583 (local address 127.0.0.1:47330)
0810 23:01:46.621857 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.621868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:46.621958 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:46.622191 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:46.622194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:46.622209 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:46.622297 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:46.622303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.623083 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.623086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:46.624253 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:46.624263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.625066 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.625071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:46.625165 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:46.625194 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:01:46.625279 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:01:46.625355 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":167,"thread_start_us":93,"threads_started":1}
I20260810 23:01:46.625553 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.621115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.15.56.2:34583)
0810 23:01:46.621387 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:46.621391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:46.621947 (+ 556us) server_negotiation.cc:407] Connection header received
0810 23:01:46.622040 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:46.622043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:46.622089 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:46.622168 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:46.622395 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.622967 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.624377 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.624927 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.625032 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:46.625104 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:46.625309 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 23:01:46.625406 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
I20260810 23:01:46.625722 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.620047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:45637 (local address 127.0.0.1:36312)
0810 23:01:46.620164 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.620197 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:46.620305 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:46.621655 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:46.621658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:46.621686 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:46.621944 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:46.621954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.623099 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.623103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:46.624519 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:46.624531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.625379 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.625382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:46.625469 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:46.625493 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:01:46.625539 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 23:01:46.625608 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":153,"spinlock_wait_cycles":3200}
I20260810 23:01:46.626010 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.620643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:40683 (local address 127.0.0.1:39070)
0810 23:01:46.621284 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.621299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:46.621427 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:46.621654 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:46.621657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:46.621685 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:46.621944 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:46.621954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.623099 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.623103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:46.624520 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:46.624531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.625519 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.625523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:46.625616 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:46.625734 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 23:01:46.625809 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 23:01:46.625877 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":131,"spinlock_wait_cycles":7168,"thread_start_us":105,"threads_started":1}
I20260810 23:01:46.626269 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.621012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.15.56.1:40683)
0810 23:01:46.621444 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:46.621449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:46.621466 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:01:46.621514 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:46.621517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:46.621563 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:46.621644 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:46.622116 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.622981 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.624670 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.625295 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.625392 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:46.625460 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:46.625848 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 23:01:46.625942 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":74,"threads_started":1}
I20260810 23:01:46.626600 28041 tablet_service.cc:1467] Processing CreateTablet for tablet 84aeadb65d244edfaaaa383934541eb5 (DEFAULT_TABLE table=test-table [id=4e4f1a206be84d1cb81c6c8a74f8fb0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:46.626812 27811 tablet_service.cc:1467] Processing CreateTablet for tablet 84aeadb65d244edfaaaa383934541eb5 (DEFAULT_TABLE table=test-table [id=4e4f1a206be84d1cb81c6c8a74f8fb0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:46.626926 28041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84aeadb65d244edfaaaa383934541eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:46.627038 27811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84aeadb65d244edfaaaa383934541eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:46.628105 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.620182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.15.56.3:45637)
0810 23:01:46.621402 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:46.621406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:46.621421 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:01:46.621478 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:46.621481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:46.621530 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:46.621646 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:46.622117 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.622952 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.624644 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.625233 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.625825 (+ 592us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:46.625890 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:46.625929 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 23:01:46.627965 (+ 2036us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1136,"thread_start_us":116,"threads_started":1}
I20260810 23:01:46.628697 28275 tablet_service.cc:1467] Processing CreateTablet for tablet 84aeadb65d244edfaaaa383934541eb5 (DEFAULT_TABLE table=test-table [id=4e4f1a206be84d1cb81c6c8a74f8fb0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:46.628983 28275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84aeadb65d244edfaaaa383934541eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:46.629349 28552 tablet_bootstrap.cc:492] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3: Bootstrap starting.
I20260810 23:01:46.629500 28555 tablet_bootstrap.cc:492] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f: Bootstrap starting.
I20260810 23:01:46.630435 28555 tablet_bootstrap.cc:654] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:46.630820 28552 tablet_bootstrap.cc:654] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:46.631858 28555 tablet_bootstrap.cc:492] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f: No bootstrap required, opened a new log
I20260810 23:01:46.631958 28555 ts_tablet_manager.cc:1397] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:46.632372 28552 tablet_bootstrap.cc:492] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3: No bootstrap required, opened a new log
I20260810 23:01:46.632351 28555 raft_consensus.cc:348] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:46.632450 28555 raft_consensus.cc:374] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:46.632472 28552 ts_tablet_manager.cc:1397] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:46.632491 28555 raft_consensus.cc:729] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166e4b15b20145908fb57ace1c03ce7f, State: Initialized, Role: FOLLOWER
I20260810 23:01:46.632632 28555 consensus_queue.cc:260] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:46.632962 28552 raft_consensus.cc:348] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:46.633062 28552 raft_consensus.cc:374] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:46.633092 28552 raft_consensus.cc:729] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a97d9983524ec28f5aef1a5176efb3, State: Initialized, Role: FOLLOWER
I20260810 23:01:46.633128 28555 ts_tablet_manager.cc:1428] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:46.633220 28552 consensus_queue.cc:260] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:46.633592 28557 tablet_bootstrap.cc:492] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab: Bootstrap starting.
I20260810 23:01:46.633615 28552 ts_tablet_manager.cc:1428] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:46.634029 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.633639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.2:50025)
0810 23:01:46.633900 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.633960 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260810 23:01:46.634756 28557 tablet_bootstrap.cc:654] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:46.635447 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.634508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.1:58535)
0810 23:01:46.635299 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.635365 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":711}
I20260810 23:01:46.636813 27919 heartbeater.cc:503] Master 127.15.56.61:39989 was elected leader, sending a full tablet report...
I20260810 23:01:46.637262 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.637014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.1:46301)
0810 23:01:46.637135 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.637193 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 23:01:46.638476 28149 heartbeater.cc:503] Master 127.15.56.61:39989 was elected leader, sending a full tablet report...
I20260810 23:01:46.639032 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.638762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.2:53527)
0810 23:01:46.638888 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.638949 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:01:46.640280 28557 tablet_bootstrap.cc:492] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab: No bootstrap required, opened a new log
I20260810 23:01:46.640386 28557 ts_tablet_manager.cc:1397] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260810 23:01:46.640846 28557 raft_consensus.cc:348] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:46.640950 28557 raft_consensus.cc:374] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:46.640971 28557 raft_consensus.cc:729] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14835bd4d1074a1194177fdf78deacab, State: Initialized, Role: FOLLOWER
I20260810 23:01:46.641113 28557 consensus_queue.cc:260] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:46.641541 28557 ts_tablet_manager.cc:1428] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:01:46.642433 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.642195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.3:47345)
0810 23:01:46.642309 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.642364 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 23:01:46.643326 28385 heartbeater.cc:503] Master 127.15.56.61:39989 was elected leader, sending a full tablet report...
I20260810 23:01:46.651279 28561 raft_consensus.cc:482] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:46.651486 28561 raft_consensus.cc:504] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:46.651969 28561 leader_election.cc:288] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14835bd4d1074a1194177fdf78deacab (127.15.56.3:45637), 55a97d9983524ec28f5aef1a5176efb3 (127.15.56.1:40683)
I20260810 23:01:46.652460 27846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84aeadb65d244edfaaaa383934541eb5" candidate_uuid: "166e4b15b20145908fb57ace1c03ce7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a97d9983524ec28f5aef1a5176efb3" is_pre_election: true
I20260810 23:01:46.652470 28307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84aeadb65d244edfaaaa383934541eb5" candidate_uuid: "166e4b15b20145908fb57ace1c03ce7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14835bd4d1074a1194177fdf78deacab" is_pre_election: true
I20260810 23:01:46.652627 28307 raft_consensus.cc:2454] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166e4b15b20145908fb57ace1c03ce7f in term 0.
I20260810 23:01:46.652627 27846 raft_consensus.cc:2454] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166e4b15b20145908fb57ace1c03ce7f in term 0.
I20260810 23:01:46.652995 27972 leader_election.cc:302] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14835bd4d1074a1194177fdf78deacab, 166e4b15b20145908fb57ace1c03ce7f, 55a97d9983524ec28f5aef1a5176efb3; no voters:
I20260810 23:01:46.653223 28561 raft_consensus.cc:2781] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:46.653299 28561 raft_consensus.cc:482] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:46.653348 28561 raft_consensus.cc:3037] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:46.654426 28561 raft_consensus.cc:504] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:46.654796 28561 leader_election.cc:288] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [CANDIDATE]: Term 1 election: Requested vote from peers 14835bd4d1074a1194177fdf78deacab (127.15.56.3:45637), 55a97d9983524ec28f5aef1a5176efb3 (127.15.56.1:40683)
I20260810 23:01:46.655153 28307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84aeadb65d244edfaaaa383934541eb5" candidate_uuid: "166e4b15b20145908fb57ace1c03ce7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14835bd4d1074a1194177fdf78deacab"
I20260810 23:01:46.655182 27846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84aeadb65d244edfaaaa383934541eb5" candidate_uuid: "166e4b15b20145908fb57ace1c03ce7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a97d9983524ec28f5aef1a5176efb3"
I20260810 23:01:46.655272 28307 raft_consensus.cc:3037] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:46.655319 27846 raft_consensus.cc:3037] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:46.656291 28307 raft_consensus.cc:2454] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166e4b15b20145908fb57ace1c03ce7f in term 1.
I20260810 23:01:46.656363 27846 raft_consensus.cc:2454] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166e4b15b20145908fb57ace1c03ce7f in term 1.
I20260810 23:01:46.656615 27972 leader_election.cc:302] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14835bd4d1074a1194177fdf78deacab, 166e4b15b20145908fb57ace1c03ce7f; no voters:
I20260810 23:01:46.656829 28561 raft_consensus.cc:2781] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:46.657064 28561 raft_consensus.cc:686] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 1 LEADER]: Becoming Leader. State: Replica: 166e4b15b20145908fb57ace1c03ce7f, State: Running, Role: LEADER
I20260810 23:01:46.657397 28561 consensus_queue.cc:237] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } } peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } }
I20260810 23:01:46.657897 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.657622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.2:45019)
0810 23:01:46.657752 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.657818 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 23:01:46.659123 27276 catalog_manager.cc:5495] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 166e4b15b20145908fb57ace1c03ce7f (127.15.56.2). New cstate: current_term: 1 leader_uuid: "166e4b15b20145908fb57ace1c03ce7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166e4b15b20145908fb57ace1c03ce7f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34583 } health_report { overall_health: HEALTHY } } }
I20260810 23:01:46.675789 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.672747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.15.56.2:34583)
0810 23:01:46.672891 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:46.672896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:46.672911 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:01:46.672964 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:46.672967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:46.673018 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:46.673080 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:46.673332 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.673862 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.674720 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:46.674959 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:46.675049 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:46.675300 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:46.675383 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:46.675443 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:46.675544 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:01:46.675607 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:01:46.678095 28307 raft_consensus.cc:1261] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab [term 1 FOLLOWER]: Refusing update from remote peer 166e4b15b20145908fb57ace1c03ce7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:46.678095 27846 raft_consensus.cc:1261] T 84aeadb65d244edfaaaa383934541eb5 P 55a97d9983524ec28f5aef1a5176efb3 [term 1 FOLLOWER]: Refusing update from remote peer 166e4b15b20145908fb57ace1c03ce7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:46.678587 28561 consensus_queue.cc:1035] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a97d9983524ec28f5aef1a5176efb3" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:46.678761 28567 consensus_queue.cc:1035] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "14835bd4d1074a1194177fdf78deacab" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:46.680562 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.680269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.3:48947)
0810 23:01:46.680399 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.680468 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 23:01:46.680977 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.680589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.1:49491)
0810 23:01:46.680838 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.680897 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260810 23:01:46.683517 28576 mvcc.cc:205] Tried to move back new op lower bound from 7317106305749012480 to 7317106305670307840. Current Snapshot: MvccSnapshot[applied={T|T < 7317106305749012480}]
I20260810 23:01:46.683706 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.683445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.2:44851)
0810 23:01:46.683547 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.683605 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 23:01:46.684600 28521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.684353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.3:45433)
0810 23:01:46.684480 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.684531 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 23:01:46.685622 28578 mvcc.cc:205] Tried to move back new op lower bound from 7317106305749012480 to 7317106305670307840. Current Snapshot: MvccSnapshot[applied={T|T < 7317106305749012480}]
I20260810 23:01:46.686489 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.686274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.2:48939)
0810 23:01:46.686367 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.686423 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 23:01:46.688566 27278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:46.688766 27278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:46.690361 28522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.690006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40785 (local address 127.15.56.1:40671)
0810 23:01:46.690211 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.690277 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260810 23:01:46.692638 27278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78fe7d9db0874fccbd42624a4e89c37f: Sending DeleteTablet for 3 replicas of tablet 84aeadb65d244edfaaaa383934541eb5
I20260810 23:01:46.693257 27811 tablet_service.cc:1514] Processing DeleteTablet for tablet 84aeadb65d244edfaaaa383934541eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:46 UTC) from {username='oryx', principal='oryx/127.15.56.61@KRBTEST.COM'} at 127.0.0.1:39070
I20260810 23:01:46.693392 28275 tablet_service.cc:1514] Processing DeleteTablet for tablet 84aeadb65d244edfaaaa383934541eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:46 UTC) from {username='oryx', principal='oryx/127.15.56.61@KRBTEST.COM'} at 127.0.0.1:36312
I20260810 23:01:46.693578 27810 tablet_service.cc:1514] Processing DeleteTablet for tablet 84aeadb65d244edfaaaa383934541eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:46 UTC) from {username='oryx', principal='oryx/127.15.56.61@KRBTEST.COM'} at 127.0.0.1:39070
I20260810 23:01:46.693720 28594 tablet_replica.cc:330] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab: stopping tablet replica
I20260810 23:01:46.693858 28594 raft_consensus.cc:2229] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:46.693881 28041 tablet_service.cc:1514] Processing DeleteTablet for tablet 84aeadb65d244edfaaaa383934541eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:46 UTC) from {username='oryx', principal='oryx/127.15.56.61@KRBTEST.COM'} at 127.0.0.1:47330
I20260810 23:01:46.693988 28594 raft_consensus.cc:2258] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:46.694242 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 27549
I20260810 23:01:46.694242 28595 tablet_replica.cc:330] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f: stopping tablet replica
I20260810 23:01:46.705971 28595 raft_consensus.cc:2229] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:46.706136 28594 ts_tablet_manager.cc:1905] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:46.706470 28595 raft_consensus.cc:2258] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:46.707458 28595 ts_tablet_manager.cc:1905] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:46.709501 28594 ts_tablet_manager.cc:1918] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:46.709589 28594 log.cc:1196] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/84aeadb65d244edfaaaa383934541eb5
I20260810 23:01:46.709659 28595 ts_tablet_manager.cc:1918] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:46.709772 28595 log.cc:1196] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/84aeadb65d244edfaaaa383934541eb5
I20260810 23:01:46.709820 28594 ts_tablet_manager.cc:1939] T 84aeadb65d244edfaaaa383934541eb5 P 14835bd4d1074a1194177fdf78deacab: Deleting consensus metadata
I20260810 23:01:46.710117 28595 ts_tablet_manager.cc:1939] T 84aeadb65d244edfaaaa383934541eb5 P 166e4b15b20145908fb57ace1c03ce7f: Deleting consensus metadata
I20260810 23:01:46.710673 27254 catalog_manager.cc:4841] TS 14835bd4d1074a1194177fdf78deacab (127.15.56.3:45637): tablet 84aeadb65d244edfaaaa383934541eb5 (table test-table [id=4e4f1a206be84d1cb81c6c8a74f8fb0b]) successfully deleted
I20260810 23:01:46.710825 27256 catalog_manager.cc:4841] TS 166e4b15b20145908fb57ace1c03ce7f (127.15.56.2:34583): tablet 84aeadb65d244edfaaaa383934541eb5 (table test-table [id=4e4f1a206be84d1cb81c6c8a74f8fb0b]) successfully deleted
I20260810 23:01:46.716440 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 27928
I20260810 23:01:46.719122 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:46.716590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:40683 (local address 127.0.0.1:39072)
0810 23:01:46.718941 (+ 2351us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:46.719010 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:40683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2238}
W20260810 23:01:46.719381 27252 catalog_manager.cc:4584] TS 55a97d9983524ec28f5aef1a5176efb3 (127.15.56.1:40683): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84aeadb65d244edfaaaa383934541eb5: Network error: Client connection negotiation failed: client connection to 127.15.56.1:40683: connect: Connection refused (error 111)
I20260810 23:01:46.730484 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 28157
I20260810 23:01:46.746948 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 27051
I20260810 23:01:46.763942 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 27237
2026-08-10T23:01:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7299 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:46 dist-test-slave-6bbx krb5kdc[28599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:46 dist-test-slave-6bbx krb5kdc[28599](info): set up 2 sockets
Aug 10 23:01:46 dist-test-slave-6bbx krb5kdc[28599](info): commencing operation
krb5kdc: starting...
W20260810 23:01:47.936501 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:48 dist-test-slave-6bbx krb5kdc[28599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.62.keytab.
Entry for principal oryx/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.62.keytab.
I20260810 23:01:48.066656 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43055
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.62.keytab
--principal=oryx/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:43055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:48.182960 28620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:48.183244 28620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:48.183300 28620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:48.187299 28620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:48.187371 28620 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:01:48.187392 28620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:48.187415 28620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:48.187436 28620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:48.187466 28620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:48.192878 28620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:43055
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.62.keytab
--principal=oryx/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:48.194007 28620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:48.195073 28620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:48.204505 28629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:48.204524 28631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:48.204483 28628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:48.204505 28620 server_base.cc:1034] running on GCE node
I20260810 23:01:48.205435 28620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:48.206609 28620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:48.207762 28620 hybrid_clock.cc:648] HybridClock initialized: now 1786402908207730 us; error 39 us; skew 500 ppm
Aug 10 23:01:48 dist-test-slave-6bbx krb5kdc[28599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402908, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:48.210624 28620 init.cc:377] Logged in from keytab as oryx/127.15.56.62@KRBTEST.COM (short username oryx)
I20260810 23:01:48.211604 28620 webserver.cc:460] Webserver started at http://127.15.56.62:36501/ using document root <none> and password file <none>
I20260810 23:01:48.212208 28620 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:48.212280 28620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:48.212500 28620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:48.214702 28620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "cdae613b93a24f13a69b59b75f18988a"
format_stamp: "Formatted at 2026-08-10 23:01:48 on dist-test-slave-6bbx"
server_key: "8eadd2ca250e5373255a90c39fd1ab02"
server_key_iv: "5b974e1d70c8bb079382a3c62cde2d26"
server_key_version: "encryptionkey@0"
I20260810 23:01:48.215277 28620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "cdae613b93a24f13a69b59b75f18988a"
format_stamp: "Formatted at 2026-08-10 23:01:48 on dist-test-slave-6bbx"
server_key: "8eadd2ca250e5373255a90c39fd1ab02"
server_key_iv: "5b974e1d70c8bb079382a3c62cde2d26"
server_key_version: "encryptionkey@0"
I20260810 23:01:48.219615 28620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 23:01:48.222523 28643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:48.224287 28620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:48.224463 28620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "cdae613b93a24f13a69b59b75f18988a"
format_stamp: "Formatted at 2026-08-10 23:01:48 on dist-test-slave-6bbx"
server_key: "8eadd2ca250e5373255a90c39fd1ab02"
server_key_iv: "5b974e1d70c8bb079382a3c62cde2d26"
server_key_version: "encryptionkey@0"
I20260810 23:01:48.224593 28620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:48.259008 28620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:48.262322 28620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:48.262524 28620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:48.270632 28620 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:43055
I20260810 23:01:48.270654 28750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:43055 every 8 connection(s)
I20260810 23:01:48.271773 28620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:48.274501 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 28620
I20260810 23:01:48.274631 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:48.274972 15584 external_mini_cluster.cc:1428] Setting key a487f8e00f2479590f70bae9b5fb8128
I20260810 23:01:48.275228 28751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:48.282512 28751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a: Bootstrap starting.
Aug 10 23:01:48 dist-test-slave-6bbx krb5kdc[28599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:48.285459 28751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:48.286607 28751 log.cc:824] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:48.289248 28751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a: No bootstrap required, opened a new log
I20260810 23:01:48.290602 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:48.276547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.15.56.62:43055)
0810 23:01:48.276987 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:48.277003 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:48.277041 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:01:48.277950 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:48.277971 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:48.278327 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:48.278620 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:48.279612 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.280703 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.281857 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.282351 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.282467 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:48.285377 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:48.285384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:48.285400 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:48.285432 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:48.287340 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:48.287982 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:48.287989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:48.287995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:48.288131 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:48.288440 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:48.288443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:48.288444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:48.288918 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:48.289087 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:48.289433 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 23:01:48.289796 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":131,"threads_started":1}
I20260810 23:01:48.292861 28751 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdae613b93a24f13a69b59b75f18988a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43055 } }
I20260810 23:01:48.293105 28751 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:48.293215 28751 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdae613b93a24f13a69b59b75f18988a, State: Initialized, Role: FOLLOWER
I20260810 23:01:48.293643 28751 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdae613b93a24f13a69b59b75f18988a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43055 } }
I20260810 23:01:48.293782 28751 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:48.293848 28751 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:48.293951 28751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:48.294966 28751 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdae613b93a24f13a69b59b75f18988a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43055 } }
I20260810 23:01:48.295346 28751 leader_election.cc:302] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdae613b93a24f13a69b59b75f18988a; no voters:
I20260810 23:01:48.295719 28751 leader_election.cc:288] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:48.295883 28758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:48.296124 28758 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [term 1 LEADER]: Becoming Leader. State: Replica: cdae613b93a24f13a69b59b75f18988a, State: Running, Role: LEADER
I20260810 23:01:48.296475 28758 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdae613b93a24f13a69b59b75f18988a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43055 } }
I20260810 23:01:48.297044 28751 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:48.298394 28760 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdae613b93a24f13a69b59b75f18988a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdae613b93a24f13a69b59b75f18988a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43055 } } }
I20260810 23:01:48.298434 28761 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdae613b93a24f13a69b59b75f18988a. Latest consensus state: current_term: 1 leader_uuid: "cdae613b93a24f13a69b59b75f18988a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdae613b93a24f13a69b59b75f18988a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43055 } } }
I20260810 23:01:48.298584 28760 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:48.298632 28761 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:48.299016 28773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:48.302466 28773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:48.308698 28773 catalog_manager.cc:1347] Generated new cluster ID: 68ca8d6e5dd84f3ab7128d641b8d72c3
I20260810 23:01:48.308794 28773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:48.334758 28773 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:48.335887 28773 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:48.348448 28773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a: Generated new TSK 0
I20260810 23:01:48.349293 28773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab.
Entry for principal oryx/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab.
I20260810 23:01:48.386919 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43055
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab
--principal=oryx/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:48.509382 28787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:48.509684 28787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:48.509730 28787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:48.513926 28787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:48.514002 28787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:48.514043 28787 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:01:48.514079 28787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:48.519702 28787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab
--principal=oryx/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.56.62:43055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:48.520910 28787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:48.521998 28787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:48.532081 28795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:48.532173 28787 server_base.cc:1034] running on GCE node
W20260810 23:01:48.532073 28796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:48.532066 28799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:48.532850 28787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:48.533524 28787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:48.534694 28787 hybrid_clock.cc:648] HybridClock initialized: now 1786402908534681 us; error 38 us; skew 500 ppm
Aug 10 23:01:48 dist-test-slave-6bbx krb5kdc[28599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402908, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:48.537607 28787 init.cc:377] Logged in from keytab as oryx/127.15.56.1@KRBTEST.COM (short username oryx)
I20260810 23:01:48.538555 28787 webserver.cc:460] Webserver started at http://127.15.56.1:38107/ using document root <none> and password file <none>
I20260810 23:01:48.539127 28787 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:48.539199 28787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:48.539429 28787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:48.541632 28787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "e008090f5cfe49c2a4742178498ee4ee"
format_stamp: "Formatted at 2026-08-10 23:01:48 on dist-test-slave-6bbx"
server_key: "d143c1980fe58032784a38247a580e2c"
server_key_iv: "625a4f0179313f91c458ca5a41fa6081"
server_key_version: "encryptionkey@0"
I20260810 23:01:48.542209 28787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "e008090f5cfe49c2a4742178498ee4ee"
format_stamp: "Formatted at 2026-08-10 23:01:48 on dist-test-slave-6bbx"
server_key: "d143c1980fe58032784a38247a580e2c"
server_key_iv: "625a4f0179313f91c458ca5a41fa6081"
server_key_version: "encryptionkey@0"
I20260810 23:01:48.546229 28787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:01:48.548962 28808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:48.550390 28787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:01:48.550522 28787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "e008090f5cfe49c2a4742178498ee4ee"
format_stamp: "Formatted at 2026-08-10 23:01:48 on dist-test-slave-6bbx"
server_key: "d143c1980fe58032784a38247a580e2c"
server_key_iv: "625a4f0179313f91c458ca5a41fa6081"
server_key_version: "encryptionkey@0"
I20260810 23:01:48.550637 28787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:48.563004 28787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:48.565972 28787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:48.566159 28787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:48.567752 28787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:48.567816 28787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:48.567874 28787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:48.567914 28787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:01:48 dist-test-slave-6bbx krb5kdc[28599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402908, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:48.578676 28787 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:41923
I20260810 23:01:48.578750 28998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:41923 every 8 connection(s)
I20260810 23:01:48.579845 28787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:48.584131 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:48.568670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:59441 (local address 127.15.56.62:43055)
0810 23:01:48.568798 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:48.568803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:48.569914 (+ 1111us) server_negotiation.cc:407] Connection header received
0810 23:01:48.570927 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:48.570932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:48.570996 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:48.571054 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:48.572671 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.573516 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.574942 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.575192 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.575303 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:48.578633 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:48.578638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:48.578640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:48.578677 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:48.581402 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:48.581955 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:48.581959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:48.581961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:48.582091 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:48.582532 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:48.582535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:48.582537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:48.582794 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:48.582902 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:48.583803 (+ 901us) server_negotiation.cc:299] Negotiation successful
0810 23:01:48.583955 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:01:48.585063 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:48.569208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43055 (local address 127.15.56.1:59441)
0810 23:01:48.569735 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:48.569782 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:48.570659 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:48.571433 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:48.571439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:48.571862 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:48.572498 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:48.572523 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.573663 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.573667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:48.574801 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:48.574814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.575371 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.575375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:48.575609 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:48.576812 (+ 1203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:48.576833 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:48.578379 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:48.581536 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:48.581541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:48.581554 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:48.581834 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:48.582220 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:48.582223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:48.582225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:48.582430 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:48.582909 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:48.582916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:48.583241 (+ 325us) client_negotiation.cc:769] Sending connection context
0810 23:01:48.583483 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 23:01:48.583907 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":132,"threads_started":1}
I20260810 23:01:48.585136 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 28787
I20260810 23:01:48.585275 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:48.585637 15584 external_mini_cluster.cc:1428] Setting key fb69ebb225cfaa185260120e50722406
I20260810 23:01:48.590376 28999 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43055
I20260810 23:01:48.590909 28999 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:48.591495 28999 heartbeater.cc:511] Master 127.15.56.62:43055 requested a full tablet report, sending...
I20260810 23:01:48.593307 28677 ts_manager.cc:194] Registered new tserver with Master: e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923)
I20260810 23:01:48.594777 28677 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:59441
I20260810 23:01:48.598568 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:48.590583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43055 (local address 127.15.56.1:48739)
0810 23:01:48.591053 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:48.591067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:48.591185 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:48.591455 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:48.591458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:48.591713 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:48.591875 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:48.591884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.593002 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.593005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:48.593863 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:48.593875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.594419 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.594422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:48.594522 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:48.595080 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:48.595093 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:48.595476 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:48.597081 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:48.597084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:48.597086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:48.597299 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:48.597579 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:48.597582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:48.597584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:48.597685 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:48.598108 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:48.598110 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:48.598195 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 23:01:48.598297 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:01:48.598425 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":110,"threads_started":1}
I20260810 23:01:48.598747 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:48.590642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:48739 (local address 127.15.56.62:43055)
0810 23:01:48.590782 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:48.590798 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:48.591153 (+ 355us) server_negotiation.cc:407] Connection header received
0810 23:01:48.591271 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:48.591274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:48.591333 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:48.591426 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:48.592022 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.592828 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.593975 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.594298 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.594408 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:48.595583 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:48.595587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:48.595590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:48.595614 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:48.596988 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:48.597400 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:48.597403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:48.597406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:48.597486 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:48.597775 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:48.597779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:48.597781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:48.597973 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:48.598065 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:48.598404 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 23:01:48.598546 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for oryx/127.15.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab.
Entry for principal oryx/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab.
I20260810 23:01:48.623231 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43055
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab
--principal=oryx/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:48.750442 29012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:48.750854 29012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:48.750931 29012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:48.757002 29012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:48.757100 29012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:48.757174 29012 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:01:48.757237 29012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:48.765502 29012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab
--principal=oryx/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.56.62:43055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:48.767113 29012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:48.768656 29012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:48.780385 29022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:48.780359 29024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:48.780405 29018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:48.780560 29012 server_base.cc:1034] running on GCE node
I20260810 23:01:48.781023 29012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:48.781715 29012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:48.782877 29012 hybrid_clock.cc:648] HybridClock initialized: now 1786402908782856 us; error 29 us; skew 500 ppm
Aug 10 23:01:48 dist-test-slave-6bbx krb5kdc[28599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402908, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:48.785702 29012 init.cc:377] Logged in from keytab as oryx/127.15.56.2@KRBTEST.COM (short username oryx)
I20260810 23:01:48.786638 29012 webserver.cc:460] Webserver started at http://127.15.56.2:43305/ using document root <none> and password file <none>
I20260810 23:01:48.787184 29012 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:48.787245 29012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:48.787437 29012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:48.789665 29012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "f5fe40da1f8f45ffa394c886d2b99122"
format_stamp: "Formatted at 2026-08-10 23:01:48 on dist-test-slave-6bbx"
server_key: "b8fb1722eb146373ab837175fc7d5f65"
server_key_iv: "598d5651a9ee284993c55b46f09c68a7"
server_key_version: "encryptionkey@0"
I20260810 23:01:48.790222 29012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "f5fe40da1f8f45ffa394c886d2b99122"
format_stamp: "Formatted at 2026-08-10 23:01:48 on dist-test-slave-6bbx"
server_key: "b8fb1722eb146373ab837175fc7d5f65"
server_key_iv: "598d5651a9ee284993c55b46f09c68a7"
server_key_version: "encryptionkey@0"
I20260810 23:01:48.793988 29012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260810 23:01:48.796419 29034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:48.797725 29012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:01:48.797851 29012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "f5fe40da1f8f45ffa394c886d2b99122"
format_stamp: "Formatted at 2026-08-10 23:01:48 on dist-test-slave-6bbx"
server_key: "b8fb1722eb146373ab837175fc7d5f65"
server_key_iv: "598d5651a9ee284993c55b46f09c68a7"
server_key_version: "encryptionkey@0"
I20260810 23:01:48.797966 29012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:48.814460 29012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:48.817624 29012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:48.817785 29012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:48.819303 29012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:48.819360 29012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:48.819404 29012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:48.819433 29012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:01:48 dist-test-slave-6bbx krb5kdc[28599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402908, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:48.830044 29012 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:35545
I20260810 23:01:48.830073 29212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:35545 every 8 connection(s)
I20260810 23:01:48.831069 29012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:48.832055 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 29012
I20260810 23:01:48.832167 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:48.832491 15584 external_mini_cluster.cc:1428] Setting key 92d13d08c13e495981a95b5fd657754f
I20260810 23:01:48.833001 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:48.819977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:36563 (local address 127.15.56.62:43055)
0810 23:01:48.820122 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:48.820127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:48.821050 (+ 923us) server_negotiation.cc:407] Connection header received
0810 23:01:48.822080 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:48.822084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:48.822152 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:48.822213 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:48.823615 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.824212 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.825577 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.825871 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.825992 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:48.829093 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:48.829096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:48.829099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:48.829126 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:48.830619 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:48.831194 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:48.831197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:48.831199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:48.831306 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:48.831736 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:48.831739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:48.831741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:48.832008 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:48.832113 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:48.832710 (+ 597us) server_negotiation.cc:299] Negotiation successful
0810 23:01:48.832846 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:01:48.833925 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:48.820414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43055 (local address 127.15.56.2:36563)
0810 23:01:48.820884 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:48.820935 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:48.821847 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:48.822630 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:48.822637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:48.822961 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:48.823470 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:48.823491 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.824349 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.824352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:48.825457 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:48.825468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.826034 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.826038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:48.826229 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:48.826881 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:48.826905 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:48.828896 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:48.830734 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:48.830740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:48.830756 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:48.831096 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:48.831400 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:48.831403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:48.831405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:48.831591 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:48.832123 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:48.832130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:48.832459 (+ 329us) client_negotiation.cc:769] Sending connection context
0810 23:01:48.832672 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 23:01:48.833020 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260810 23:01:48.835189 29214 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43055
I20260810 23:01:48.835353 29214 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:48.835980 29214 heartbeater.cc:511] Master 127.15.56.62:43055 requested a full tablet report, sending...
I20260810 23:01:48.837014 28678 ts_manager.cc:194] Registered new tserver with Master: f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2:35545)
I20260810 23:01:48.837702 28678 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:36563
WARNING: no policy specified for oryx/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:01:48.845829 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:48.837541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:36201 (local address 127.15.56.62:43055)
0810 23:01:48.837663 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:48.837668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:48.838052 (+ 384us) server_negotiation.cc:407] Connection header received
0810 23:01:48.838218 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:48.838221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:48.838265 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:48.838312 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:48.839013 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.839795 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.840984 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.841200 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.841287 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:48.842628 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:48.842631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:48.842633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:48.842657 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:48.844170 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:48.844576 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:48.844580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:48.844582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:48.844665 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:48.844917 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:48.844920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:48.844922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:48.845186 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:48.845278 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:48.845545 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 23:01:48.845659 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:01:48.845841 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:48.837483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43055 (local address 127.15.56.2:36201)
0810 23:01:48.837972 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:48.837987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:48.838123 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:48.838448 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:48.838450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:48.838686 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:48.838876 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:48.838886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.839956 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.839960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:48.840849 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:48.840864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:48.841324 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:48.841328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:48.841433 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:48.841991 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:48.842007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:48.842468 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:48.844266 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:48.844277 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:48.844279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:48.844484 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:48.844747 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:48.844750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:48.844752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:48.844834 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:48.845289 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:48.845292 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:48.845388 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 23:01:48.845476 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:01:48.845635 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":1,"thread_start_us":232,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab.
Entry for principal oryx/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab.
I20260810 23:01:48.863215 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43055
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab
--principal=oryx/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:48.981427 29225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:48.981740 29225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:48.981786 29225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:48.985865 29225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:48.985927 29225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:48.985966 29225 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:01:48.985998 29225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:48.991376 29225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab
--principal=oryx/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.56.62:43055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:48.992501 29225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:48.993541 29225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:49.003857 29236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:49.003857 29233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:49.003859 29232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:49.004263 29225 server_base.cc:1034] running on GCE node
I20260810 23:01:49.004705 29225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:49.005375 29225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:49.006567 29225 hybrid_clock.cc:648] HybridClock initialized: now 1786402909006551 us; error 25 us; skew 500 ppm
Aug 10 23:01:49 dist-test-slave-6bbx krb5kdc[28599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402909, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:49.009575 29225 init.cc:377] Logged in from keytab as oryx/127.15.56.3@KRBTEST.COM (short username oryx)
I20260810 23:01:49.010591 29225 webserver.cc:460] Webserver started at http://127.15.56.3:44313/ using document root <none> and password file <none>
I20260810 23:01:49.011209 29225 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:49.011279 29225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:49.011497 29225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:49.013785 29225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "25dd4635913c4fd7bb24e23fe6d32b70"
format_stamp: "Formatted at 2026-08-10 23:01:49 on dist-test-slave-6bbx"
server_key: "6c7bdd5dad8e3abe57903dac7734f103"
server_key_iv: "a6bbfea5d2029f13bb3bef53f9c815e7"
server_key_version: "encryptionkey@0"
I20260810 23:01:49.014353 29225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "25dd4635913c4fd7bb24e23fe6d32b70"
format_stamp: "Formatted at 2026-08-10 23:01:49 on dist-test-slave-6bbx"
server_key: "6c7bdd5dad8e3abe57903dac7734f103"
server_key_iv: "a6bbfea5d2029f13bb3bef53f9c815e7"
server_key_version: "encryptionkey@0"
I20260810 23:01:49.018559 29225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 23:01:49.021337 29245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:49.022689 29225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:01:49.022825 29225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "25dd4635913c4fd7bb24e23fe6d32b70"
format_stamp: "Formatted at 2026-08-10 23:01:49 on dist-test-slave-6bbx"
server_key: "6c7bdd5dad8e3abe57903dac7734f103"
server_key_iv: "a6bbfea5d2029f13bb3bef53f9c815e7"
server_key_version: "encryptionkey@0"
I20260810 23:01:49.022944 29225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:49.046510 29225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:49.050231 29225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:49.050442 29225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:49.052501 29225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:49.052589 29225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:49.052654 29225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:49.052681 29225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:49.062832 29225 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:40791
I20260810 23:01:49.062866 29428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:40791 every 8 connection(s)
Aug 10 23:01:49 dist-test-slave-6bbx krb5kdc[28599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402909, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:49.063889 29225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:49.066968 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.053365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:59009 (local address 127.15.56.62:43055)
0810 23:01:49.053554 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.053559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.054605 (+ 1046us) server_negotiation.cc:407] Connection header received
0810 23:01:49.055880 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.055884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.055947 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.056030 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:49.057651 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.058503 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.060193 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.060548 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.060671 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.063525 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:49.063529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:49.063531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:49.063562 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:49.065106 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:49.065564 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:49.065567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:49.065568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:49.065649 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:49.065924 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:49.065925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:49.065926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:49.066093 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:49.066164 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.066727 (+ 563us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.066846 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 23:01:49.067739 29257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.053896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43055 (local address 127.15.56.3:59009)
0810 23:01:49.054415 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.054472 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.055596 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.056243 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.056253 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.056712 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:49.057479 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.057503 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.058650 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.058653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.060065 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.060077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.060705 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.060710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.060912 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.061697 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:49.061720 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:49.063346 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:49.065197 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:49.065201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:49.065232 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:49.065476 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:49.065718 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:49.065720 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:49.065721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:49.065857 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:49.066187 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:49.066193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:49.066451 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.066613 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.066870 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260810 23:01:49.068912 29429 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43055
I20260810 23:01:49.069061 29429 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:49.069643 29429 heartbeater.cc:511] Master 127.15.56.62:43055 requested a full tablet report, sending...
I20260810 23:01:49.070852 28677 ts_manager.cc:194] Registered new tserver with Master: 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3:40791)
I20260810 23:01:49.071539 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 29225
I20260810 23:01:49.071544 28677 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:59009
I20260810 23:01:49.071698 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:49.072041 15584 external_mini_cluster.cc:1428] Setting key 4651f77787a410947dba17865d1edb29
I20260810 23:01:49.074004 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:01:49.080559 29435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.071302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43055 (local address 127.15.56.3:49123)
0810 23:01:49.071973 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.071989 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.072095 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.072401 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.072404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.072623 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:49.072765 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.072773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.073812 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.073815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.074475 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.074486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.075082 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.075085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.075205 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.075725 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:49.075739 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:49.076159 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:49.078025 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:49.078029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:49.078031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:49.078305 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:49.079257 (+ 952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:49.079260 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:49.079262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:49.079372 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:49.079975 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:49.079978 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:49.080082 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.080181 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.080360 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":100,"threads_started":1}
I20260810 23:01:49.080596 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.071363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:49123 (local address 127.15.56.62:43055)
0810 23:01:49.071475 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.071479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.072063 (+ 584us) server_negotiation.cc:407] Connection header received
0810 23:01:49.072225 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.072228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.072280 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.072364 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:49.072907 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.073643 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.074605 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.074945 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.075038 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.076321 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:49.076326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:49.076328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:49.076354 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:49.077903 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:49.079029 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:49.079033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:49.079035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:49.079143 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:49.079501 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:49.079505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:49.079507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:49.079817 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:49.079898 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.080289 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.080434 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 23:01:49 dist-test-slave-6bbx krb5kdc[28599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402909, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:01:49 dist-test-slave-6bbx krb5kdc[28599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:49 dist-test-slave-6bbx krb5kdc[28599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:01:49.104686 28755 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:49.097565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39004 (local address 127.15.56.62:43055)
0810 23:01:49.097756 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.097760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.097837 (+ 77us) server_negotiation.cc:407] Connection header received
0810 23:01:49.097891 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.097894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.097947 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.098032 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:49.098927 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.099502 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.100462 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.100689 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.100791 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.104517 (+ 3726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39004: BlockingRecv error: recv got EOF from 127.0.0.1:39004 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 10 23:01:49 dist-test-slave-6bbx krb5kdc[28599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402909, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:49.114259 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.106206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.15.56.62:43055)
0810 23:01:49.106347 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.106351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.106492 (+ 141us) server_negotiation.cc:407] Connection header received
0810 23:01:49.106640 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.106644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.106692 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.106757 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:49.107403 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.108032 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.108853 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.109082 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.109176 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.111509 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:49.111512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:49.111514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:49.111543 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:49.112705 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:49.113096 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:49.113099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:49.113100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:49.113181 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:49.113436 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:49.113440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:49.113442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:49.113626 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:49.113753 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.113980 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.114095 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:01:49.117404 28677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:01:49.120509 28677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:49.135715 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.129968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.15.56.3:40791)
0810 23:01:49.130326 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.130339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.130915 (+ 576us) server_negotiation.cc:407] Connection header received
0810 23:01:49.131121 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.131126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.131309 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.131446 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:49.131877 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.132645 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.134245 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.134954 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.135056 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.135249 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.135422 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.135525 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
I20260810 23:01:49.135720 29469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.129595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:40791 (local address 127.0.0.1:53580)
0810 23:01:49.130816 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.130829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.130987 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.131632 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.131634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.131650 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:49.131776 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.131783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.132763 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.132772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.134114 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.134125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.135069 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.135072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.135192 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.135236 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.135383 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.135520 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1158,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260810 23:01:49.136417 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.129509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:35545 (local address 127.0.0.1:50490)
0810 23:01:49.130461 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.130514 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.130656 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.131476 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.131484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.131508 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:49.131656 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.131691 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.133196 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.133200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.134346 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.134357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.135715 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.135718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.135827 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.136099 (+ 272us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.136181 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.136272 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":87,"thread_start_us":119,"threads_started":1}
I20260810 23:01:49.136463 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.129664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.15.56.2:35545)
0810 23:01:49.130970 (+ 1306us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.130982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.131002 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:01:49.131082 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.131087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.131306 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.131454 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:49.131820 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.133043 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.134475 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.135394 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.135479 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.135639 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.136230 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.136323 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1200,"thread_start_us":130,"threads_started":1}
I20260810 23:01:49.136879 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.130300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:41923 (local address 127.0.0.1:45730)
0810 23:01:49.130954 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.130967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.131059 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.131732 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.131735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.131753 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:49.131848 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.131854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.133158 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.133162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.134371 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.134380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.135714 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.135718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.136120 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.136139 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.136196 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.136746 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":645,"thread_start_us":90,"threads_started":1}
I20260810 23:01:49.137327 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.130470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.15.56.1:41923)
0810 23:01:49.130767 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.130774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.131154 (+ 380us) server_negotiation.cc:407] Connection header received
0810 23:01:49.131240 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.131245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.131436 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.131578 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:49.131948 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.133036 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.134494 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.135367 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.135477 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.135623 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.136301 (+ 678us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.136403 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260810 23:01:49.139042 29108 tablet_service.cc:1467] Processing CreateTablet for tablet 56deba6a50b9421891c33a43d0becda2 (DEFAULT_TABLE table=test-table [id=30baef939c4d4e7790bab21b323bf5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.139146 29327 tablet_service.cc:1467] Processing CreateTablet for tablet 56deba6a50b9421891c33a43d0becda2 (DEFAULT_TABLE table=test-table [id=30baef939c4d4e7790bab21b323bf5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.140143 29108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56deba6a50b9421891c33a43d0becda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.140322 29327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56deba6a50b9421891c33a43d0becda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.140130 28888 tablet_service.cc:1467] Processing CreateTablet for tablet 56deba6a50b9421891c33a43d0becda2 (DEFAULT_TABLE table=test-table [id=30baef939c4d4e7790bab21b323bf5d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.141153 28888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56deba6a50b9421891c33a43d0becda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.147257 29473 tablet_bootstrap.cc:492] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122: Bootstrap starting.
I20260810 23:01:49.149053 29476 tablet_bootstrap.cc:492] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee: Bootstrap starting.
I20260810 23:01:49.149053 29474 tablet_bootstrap.cc:492] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70: Bootstrap starting.
I20260810 23:01:49.149614 29473 tablet_bootstrap.cc:654] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.150362 29473 log.cc:824] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:49.151314 29474 tablet_bootstrap.cc:654] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.151305 29476 tablet_bootstrap.cc:654] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.152261 29476 log.cc:824] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:49.152261 29474 log.cc:824] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:49.153081 29473 tablet_bootstrap.cc:492] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122: No bootstrap required, opened a new log
I20260810 23:01:49.153326 29473 ts_tablet_manager.cc:1397] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:01:49.154423 29476 tablet_bootstrap.cc:492] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee: No bootstrap required, opened a new log
I20260810 23:01:49.154639 29476 ts_tablet_manager.cc:1397] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:01:49.154760 29474 tablet_bootstrap.cc:492] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70: No bootstrap required, opened a new log
I20260810 23:01:49.154995 29474 ts_tablet_manager.cc:1397] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:01:49.156759 29473 raft_consensus.cc:348] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } }
I20260810 23:01:49.157006 29473 raft_consensus.cc:374] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.157064 29473 raft_consensus.cc:729] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fe40da1f8f45ffa394c886d2b99122, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.157526 29473 consensus_queue.cc:260] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } }
I20260810 23:01:49.158215 29214 heartbeater.cc:503] Master 127.15.56.62:43055 was elected leader, sending a full tablet report...
I20260810 23:01:49.158996 29474 raft_consensus.cc:348] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } }
I20260810 23:01:49.159278 29474 raft_consensus.cc:374] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.159377 29474 raft_consensus.cc:729] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25dd4635913c4fd7bb24e23fe6d32b70, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.159271 29476 raft_consensus.cc:348] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } }
I20260810 23:01:49.159526 29476 raft_consensus.cc:374] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.159579 29476 raft_consensus.cc:729] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e008090f5cfe49c2a4742178498ee4ee, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.159896 29474 consensus_queue.cc:260] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } }
I20260810 23:01:49.160297 29476 consensus_queue.cc:260] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } }
I20260810 23:01:49.160687 29429 heartbeater.cc:503] Master 127.15.56.62:43055 was elected leader, sending a full tablet report...
I20260810 23:01:49.161000 29474 ts_tablet_manager.cc:1428] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:01:49.161182 29476 ts_tablet_manager.cc:1428] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:01:49.161602 28999 heartbeater.cc:503] Master 127.15.56.62:43055 was elected leader, sending a full tablet report...
I20260810 23:01:49.161710 29481 raft_consensus.cc:482] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:49.161860 29481 raft_consensus.cc:504] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } }
I20260810 23:01:49.162130 29473 ts_tablet_manager.cc:1428] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260810 23:01:49.163097 29481 leader_election.cc:288] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3:40791), e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923)
I20260810 23:01:49.167388 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.163405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:40791 (local address 127.15.56.2:58827)
0810 23:01:49.163712 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.163727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.163812 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.164200 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.164203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.164225 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:49.164364 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.164373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.165325 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.165328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.166276 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.166285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.166930 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.166934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.167046 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.167067 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.167134 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.167261 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":41}
I20260810 23:01:49.167388 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.163715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:58827 (local address 127.15.56.3:40791)
0810 23:01:49.163980 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.163984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.163998 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:01:49.164044 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.164047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.164096 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.164174 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:49.164503 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.165205 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.166391 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.166816 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.166912 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.166979 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.167160 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.167242 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260810 23:01:49.168066 29357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56deba6a50b9421891c33a43d0becda2" candidate_uuid: "f5fe40da1f8f45ffa394c886d2b99122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" is_pre_election: true
I20260810 23:01:49.168434 29357 raft_consensus.cc:2454] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5fe40da1f8f45ffa394c886d2b99122 in term 0.
I20260810 23:01:49.168924 29041 leader_election.cc:302] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25dd4635913c4fd7bb24e23fe6d32b70, f5fe40da1f8f45ffa394c886d2b99122; no voters:
I20260810 23:01:49.169231 29481 raft_consensus.cc:2781] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:49.169323 29481 raft_consensus.cc:482] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:49.169369 29481 raft_consensus.cc:3037] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.170308 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.163569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:39647 (local address 127.15.56.1:41923)
0810 23:01:49.163712 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.163717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.165320 (+ 1603us) server_negotiation.cc:407] Connection header received
0810 23:01:49.165511 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.165514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.165568 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.165615 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:49.166433 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.166914 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.169163 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.169714 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.169812 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.169866 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.170113 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.170186 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:01:49.170342 29485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.163409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:41923 (local address 127.15.56.2:39647)
0810 23:01:49.165215 (+ 1806us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.165229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.165391 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.165756 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.165759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.165775 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:49.166316 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.166326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.167023 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.167026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.169029 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.169042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.169873 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.169877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.169973 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.170001 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.170083 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.170164 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1733,"mutex_wait_us":1271,"thread_start_us":118,"threads_started":1}
I20260810 23:01:49.170816 29481 raft_consensus.cc:504] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } }
I20260810 23:01:49.171033 28922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56deba6a50b9421891c33a43d0becda2" candidate_uuid: "f5fe40da1f8f45ffa394c886d2b99122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e008090f5cfe49c2a4742178498ee4ee" is_pre_election: true
I20260810 23:01:49.171233 29481 leader_election.cc:288] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [CANDIDATE]: Term 1 election: Requested vote from peers 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3:40791), e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923)
I20260810 23:01:49.171375 28922 raft_consensus.cc:2454] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5fe40da1f8f45ffa394c886d2b99122 in term 0.
I20260810 23:01:49.171568 29357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56deba6a50b9421891c33a43d0becda2" candidate_uuid: "f5fe40da1f8f45ffa394c886d2b99122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25dd4635913c4fd7bb24e23fe6d32b70"
I20260810 23:01:49.171723 29357 raft_consensus.cc:3037] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.172195 28922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56deba6a50b9421891c33a43d0becda2" candidate_uuid: "f5fe40da1f8f45ffa394c886d2b99122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e008090f5cfe49c2a4742178498ee4ee"
I20260810 23:01:49.172323 28922 raft_consensus.cc:3037] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.173151 29357 raft_consensus.cc:2454] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5fe40da1f8f45ffa394c886d2b99122 in term 1.
I20260810 23:01:49.173491 29041 leader_election.cc:302] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25dd4635913c4fd7bb24e23fe6d32b70, f5fe40da1f8f45ffa394c886d2b99122; no voters:
I20260810 23:01:49.173664 28922 raft_consensus.cc:2454] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5fe40da1f8f45ffa394c886d2b99122 in term 1.
I20260810 23:01:49.173715 29481 raft_consensus.cc:2781] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:49.173960 29481 raft_consensus.cc:686] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 1 LEADER]: Becoming Leader. State: Replica: f5fe40da1f8f45ffa394c886d2b99122, State: Running, Role: LEADER
I20260810 23:01:49.174330 29481 consensus_queue.cc:237] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } }
I20260810 23:01:49.177975 28677 catalog_manager.cc:5495] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2). New cstate: current_term: 1 leader_uuid: "f5fe40da1f8f45ffa394c886d2b99122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } health_report { overall_health: UNKNOWN } } }
I20260810 23:01:49.200399 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.196650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.15.56.2:35545)
0810 23:01:49.196781 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.196784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.196798 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:01:49.196936 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.196938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.197000 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.197085 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:49.197309 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.197886 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.198898 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.199135 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.199282 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.199492 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:49.200000 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:49.200091 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.200193 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.200247 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:01:49.206561 28922 raft_consensus.cc:1261] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Refusing update from remote peer f5fe40da1f8f45ffa394c886d2b99122: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:49.206957 29357 raft_consensus.cc:1261] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Refusing update from remote peer f5fe40da1f8f45ffa394c886d2b99122: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:49.207237 29486 consensus_queue.cc:1035] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:49.207520 29481 consensus_queue.cc:1035] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:49.214497 29493 mvcc.cc:205] Tried to move back new op lower bound from 7317106316102995968 to 7317106315980828672. Current Snapshot: MvccSnapshot[applied={T|T < 7317106316102995968}]
I20260810 23:01:49.216022 29494 mvcc.cc:205] Tried to move back new op lower bound from 7317106316102995968 to 7317106315980828672. Current Snapshot: MvccSnapshot[applied={T|T < 7317106316102995968}]
I20260810 23:01:49.217514 29495 mvcc.cc:205] Tried to move back new op lower bound from 7317106316102995968 to 7317106315980828672. Current Snapshot: MvccSnapshot[applied={T|T < 7317106316102995968 or (T in {7317106316102995968})}]
I20260810 23:01:49.223869 28678 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:49.224045 28678 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:49.226501 28678 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a: Sending DeleteTablet for 3 replicas of tablet 56deba6a50b9421891c33a43d0becda2
I20260810 23:01:49.227380 29108 tablet_service.cc:1514] Processing DeleteTablet for tablet 56deba6a50b9421891c33a43d0becda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:49 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:50490
I20260810 23:01:49.227485 29327 tablet_service.cc:1514] Processing DeleteTablet for tablet 56deba6a50b9421891c33a43d0becda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:49 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:53580
I20260810 23:01:49.227455 28888 tablet_service.cc:1514] Processing DeleteTablet for tablet 56deba6a50b9421891c33a43d0becda2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:49 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45730
I20260810 23:01:49.228143 29509 tablet_replica.cc:330] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70: stopping tablet replica
I20260810 23:01:49.228302 29510 tablet_replica.cc:330] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee: stopping tablet replica
I20260810 23:01:49.228533 29509 raft_consensus.cc:2229] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:49.228737 29510 raft_consensus.cc:2229] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:49.228930 29509 raft_consensus.cc:2258] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:49.228919 28678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:01:49.229382 28678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:49.229396 29510 raft_consensus.cc:2258] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:49.229492 29508 tablet_replica.cc:330] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122: stopping tablet replica
I20260810 23:01:49.229827 29508 raft_consensus.cc:2229] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:49.230257 29508 raft_consensus.cc:2258] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:49.231719 29509 ts_tablet_manager.cc:1905] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:49.231777 29508 ts_tablet_manager.cc:1905] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:49.234624 29508 ts_tablet_manager.cc:1918] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:49.234591 29509 ts_tablet_manager.cc:1918] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:49.234728 29508 log.cc:1196] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/56deba6a50b9421891c33a43d0becda2
I20260810 23:01:49.234735 29509 log.cc:1196] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/56deba6a50b9421891c33a43d0becda2
I20260810 23:01:49.235069 29509 ts_tablet_manager.cc:1939] T 56deba6a50b9421891c33a43d0becda2 P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting consensus metadata
I20260810 23:01:49.235136 29508 ts_tablet_manager.cc:1939] T 56deba6a50b9421891c33a43d0becda2 P f5fe40da1f8f45ffa394c886d2b99122: Deleting consensus metadata
I20260810 23:01:49.235529 28888 tablet_service.cc:1467] Processing CreateTablet for tablet aee76f9b912c4afb9d6c3ff2f20e707a (DEFAULT_TABLE table=test-table [id=b395e351d8f44dd1a985e1bdfa70816a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.235903 29108 tablet_service.cc:1467] Processing CreateTablet for tablet aee76f9b912c4afb9d6c3ff2f20e707a (DEFAULT_TABLE table=test-table [id=b395e351d8f44dd1a985e1bdfa70816a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.236166 29108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee76f9b912c4afb9d6c3ff2f20e707a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.236483 28645 catalog_manager.cc:4841] TS f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2:35545): tablet 56deba6a50b9421891c33a43d0becda2 (table test-table [id=30baef939c4d4e7790bab21b323bf5d4]) successfully deleted
I20260810 23:01:49.236644 28655 catalog_manager.cc:4841] TS 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3:40791): tablet 56deba6a50b9421891c33a43d0becda2 (table test-table [id=30baef939c4d4e7790bab21b323bf5d4]) successfully deleted
I20260810 23:01:49.237004 28888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee76f9b912c4afb9d6c3ff2f20e707a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.237015 29327 tablet_service.cc:1467] Processing CreateTablet for tablet aee76f9b912c4afb9d6c3ff2f20e707a (DEFAULT_TABLE table=test-table [id=b395e351d8f44dd1a985e1bdfa70816a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.237263 29327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee76f9b912c4afb9d6c3ff2f20e707a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.238577 29473 tablet_bootstrap.cc:492] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122: Bootstrap starting.
I20260810 23:01:49.239490 29474 tablet_bootstrap.cc:492] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70: Bootstrap starting.
I20260810 23:01:49.239627 29473 tablet_bootstrap.cc:654] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.240348 29476 tablet_bootstrap.cc:492] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee: Bootstrap starting.
I20260810 23:01:49.240499 29474 tablet_bootstrap.cc:654] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.240926 29510 ts_tablet_manager.cc:1905] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:49.241333 29476 tablet_bootstrap.cc:654] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.243323 29474 tablet_bootstrap.cc:492] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70: No bootstrap required, opened a new log
I20260810 23:01:49.243422 29474 ts_tablet_manager.cc:1397] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:01:49.243520 29476 tablet_bootstrap.cc:492] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee: No bootstrap required, opened a new log
I20260810 23:01:49.243607 29476 ts_tablet_manager.cc:1397] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:49.243695 29510 ts_tablet_manager.cc:1918] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:49.243827 29510 log.cc:1196] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/wal/wals/56deba6a50b9421891c33a43d0becda2
I20260810 23:01:49.243963 29474 raft_consensus.cc:348] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.244093 29474 raft_consensus.cc:374] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.244135 29474 raft_consensus.cc:729] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25dd4635913c4fd7bb24e23fe6d32b70, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.244230 29510 ts_tablet_manager.cc:1939] T 56deba6a50b9421891c33a43d0becda2 P e008090f5cfe49c2a4742178498ee4ee: Deleting consensus metadata
I20260810 23:01:49.244269 29474 consensus_queue.cc:260] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.244309 29476 raft_consensus.cc:348] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.244411 29476 raft_consensus.cc:374] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.244450 29476 raft_consensus.cc:729] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e008090f5cfe49c2a4742178498ee4ee, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.244549 29474 ts_tablet_manager.cc:1428] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 23:01:49.244570 29476 consensus_queue.cc:260] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.244818 29476 ts_tablet_manager.cc:1428] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:49.245800 28645 catalog_manager.cc:4841] TS e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923): tablet 56deba6a50b9421891c33a43d0becda2 (table test-table [id=30baef939c4d4e7790bab21b323bf5d4]) successfully deleted
I20260810 23:01:49.246853 29473 tablet_bootstrap.cc:492] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122: No bootstrap required, opened a new log
I20260810 23:01:49.246963 29473 ts_tablet_manager.cc:1397] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260810 23:01:49.247460 29473 raft_consensus.cc:348] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.247577 29473 raft_consensus.cc:374] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.247598 29473 raft_consensus.cc:729] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fe40da1f8f45ffa394c886d2b99122, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.247736 29473 consensus_queue.cc:260] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.248006 29473 ts_tablet_manager.cc:1428] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:01:49.289418 29482 raft_consensus.cc:482] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:49.289588 29482 raft_consensus.cc:504] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.290773 29482 leader_election.cc:288] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923), f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2:35545)
I20260810 23:01:49.295806 29257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.291189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:41923 (local address 127.15.56.3:37475)
0810 23:01:49.291361 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.291380 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.291502 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.291927 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.291929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.291947 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:49.292064 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.292071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.293033 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.293036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.294327 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.294339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.295288 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.295292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.295402 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.295428 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.295509 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.295598 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 23:01:49.295818 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.291308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:37475 (local address 127.15.56.1:41923)
0810 23:01:49.291432 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.291437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.291453 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:01:49.291647 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.291651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.291727 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.291801 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:49.292200 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.292919 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.294458 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.295062 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.295146 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.295203 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.295550 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.295649 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:01:49.296339 28922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee76f9b912c4afb9d6c3ff2f20e707a" candidate_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e008090f5cfe49c2a4742178498ee4ee" is_pre_election: true
I20260810 23:01:49.296499 28922 raft_consensus.cc:2454] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25dd4635913c4fd7bb24e23fe6d32b70 in term 0.
I20260810 23:01:49.296911 29246 leader_election.cc:302] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25dd4635913c4fd7bb24e23fe6d32b70, e008090f5cfe49c2a4742178498ee4ee; no voters:
I20260810 23:01:49.297154 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.291485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:35545 (local address 127.15.56.3:56743)
0810 23:01:49.291923 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.291937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.292057 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.292321 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.292323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.292340 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:49.292483 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.292489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.293317 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.293320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.294203 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.294214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.296700 (+ 2486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.296703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.296828 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.296855 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.296937 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.297015 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":93,"threads_started":1}
I20260810 23:01:49.297212 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.291685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:56743 (local address 127.15.56.2:35545)
0810 23:01:49.291841 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.291845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.292039 (+ 194us) server_negotiation.cc:407] Connection header received
0810 23:01:49.292162 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.292165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.292215 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.292290 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:49.292593 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.293208 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.294345 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.294959 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.295065 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.295147 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.296977 (+ 1830us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.297068 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:01:49.297283 29482 raft_consensus.cc:2781] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:49.297338 29482 raft_consensus.cc:482] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:49.297374 29482 raft_consensus.cc:3037] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.298560 29482 raft_consensus.cc:504] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.298930 29482 leader_election.cc:288] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [CANDIDATE]: Term 1 election: Requested vote from peers e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923), f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2:35545)
I20260810 23:01:49.299556 28922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee76f9b912c4afb9d6c3ff2f20e707a" candidate_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e008090f5cfe49c2a4742178498ee4ee"
I20260810 23:01:49.299734 28922 raft_consensus.cc:3037] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.299650 29140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee76f9b912c4afb9d6c3ff2f20e707a" candidate_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fe40da1f8f45ffa394c886d2b99122" is_pre_election: true
I20260810 23:01:49.299968 29140 raft_consensus.cc:2454] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25dd4635913c4fd7bb24e23fe6d32b70 in term 0.
I20260810 23:01:49.299685 29116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee76f9b912c4afb9d6c3ff2f20e707a" candidate_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fe40da1f8f45ffa394c886d2b99122"
I20260810 23:01:49.300288 29116 raft_consensus.cc:3037] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.300877 28922 raft_consensus.cc:2454] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25dd4635913c4fd7bb24e23fe6d32b70 in term 1.
I20260810 23:01:49.301162 29246 leader_election.cc:302] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25dd4635913c4fd7bb24e23fe6d32b70, e008090f5cfe49c2a4742178498ee4ee; no voters:
I20260810 23:01:49.301415 29482 raft_consensus.cc:2781] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:49.301486 29116 raft_consensus.cc:2454] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25dd4635913c4fd7bb24e23fe6d32b70 in term 1.
I20260810 23:01:49.301678 29482 raft_consensus.cc:686] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 LEADER]: Becoming Leader. State: Replica: 25dd4635913c4fd7bb24e23fe6d32b70, State: Running, Role: LEADER
I20260810 23:01:49.301986 29482 consensus_queue.cc:237] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.304610 28678 catalog_manager.cc:5495] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3). New cstate: current_term: 1 leader_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } health_report { overall_health: HEALTHY } } }
W20260810 23:01:49.314944 29430 tablet.cc:2369] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:01:49.331736 29000 tablet.cc:2369] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:01:49.333459 29215 tablet.cc:2369] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 23:01:49 dist-test-slave-6bbx krb5kdc[28599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402908, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.62@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:49.343142 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.333686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43055 (local address 127.0.0.1:39034)
0810 23:01:49.334016 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.334032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.334147 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.334430 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.334432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.334855 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:49.334962 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.334969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.336058 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.336061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.336695 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.336704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.337249 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.337251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.337315 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.337922 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:49.337935 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:49.339569 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:49.341467 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:49.341478 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:49.341484 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:49.341785 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:49.342084 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:49.342087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:49.342088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:49.342186 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:49.342623 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:49.342629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:49.342768 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.342868 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.342997 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":126,"threads_started":1}
I20260810 23:01:49.343264 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.333603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.15.56.62:43055)
0810 23:01:49.333736 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.333741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.334127 (+ 386us) server_negotiation.cc:407] Connection header received
0810 23:01:49.334235 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.334238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.334307 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.334408 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:49.335100 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.335952 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.336811 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.337131 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.337243 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.339756 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:49.339759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:49.339762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:49.339791 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:49.341328 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:49.341889 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:49.341893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:49.341895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:49.341995 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:49.342265 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:49.342275 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:49.342277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:49.342518 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:49.342617 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.342937 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.343070 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:01:49.347867 28678 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:39034:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:01:49.348686 28678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:49.354300 29108 tablet_service.cc:1467] Processing CreateTablet for tablet 2e45be9ebbd34605b7d28b117eda34cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50ea7d0510e041248f86382c22430326]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:01:49.354300 29327 tablet_service.cc:1467] Processing CreateTablet for tablet 2e45be9ebbd34605b7d28b117eda34cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50ea7d0510e041248f86382c22430326]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:01:49.354310 28888 tablet_service.cc:1467] Processing CreateTablet for tablet 2e45be9ebbd34605b7d28b117eda34cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50ea7d0510e041248f86382c22430326]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:01:49.354789 28888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e45be9ebbd34605b7d28b117eda34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.354813 29327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e45be9ebbd34605b7d28b117eda34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.354813 29108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e45be9ebbd34605b7d28b117eda34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.358469 29474 tablet_bootstrap.cc:492] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70: Bootstrap starting.
I20260810 23:01:49.358794 29473 tablet_bootstrap.cc:492] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122: Bootstrap starting.
I20260810 23:01:49.359051 29476 tablet_bootstrap.cc:492] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: Bootstrap starting.
I20260810 23:01:49.359604 29474 tablet_bootstrap.cc:654] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.359790 29473 tablet_bootstrap.cc:654] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.360208 29476 tablet_bootstrap.cc:654] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.361862 29473 tablet_bootstrap.cc:492] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122: No bootstrap required, opened a new log
I20260810 23:01:49.361963 29473 ts_tablet_manager.cc:1397] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 23:01:49.362092 29474 tablet_bootstrap.cc:492] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70: No bootstrap required, opened a new log
I20260810 23:01:49.362200 29474 ts_tablet_manager.cc:1397] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:01:49.362495 29473 raft_consensus.cc:348] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.362619 29473 raft_consensus.cc:374] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.362653 29473 raft_consensus.cc:729] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fe40da1f8f45ffa394c886d2b99122, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.362622 29474 raft_consensus.cc:348] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.362736 29474 raft_consensus.cc:374] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.362766 29474 raft_consensus.cc:729] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25dd4635913c4fd7bb24e23fe6d32b70, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.362808 29473 consensus_queue.cc:260] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.362883 29474 consensus_queue.cc:260] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.363142 29474 ts_tablet_manager.cc:1428] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:49.363098 29501 tablet_replica.cc:439] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } }
I20260810 23:01:49.363287 29501 tablet_replica.cc:442] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:49.363854 29476 tablet_bootstrap.cc:492] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: No bootstrap required, opened a new log
I20260810 23:01:49.363953 29476 ts_tablet_manager.cc:1397] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 23:01:49.363101 29473 ts_tablet_manager.cc:1428] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 23:01:49.364393 29476 raft_consensus.cc:348] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.364511 29476 raft_consensus.cc:374] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.364542 29476 raft_consensus.cc:729] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e008090f5cfe49c2a4742178498ee4ee, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.364661 29476 consensus_queue.cc:260] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.364969 29476 ts_tablet_manager.cc:1428] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 23:01:49.365010 29483 tablet_replica.cc:439] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } }
I20260810 23:01:49.365152 29483 tablet_replica.cc:442] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:49.365430 29482 tablet_replica.cc:439] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } }
I20260810 23:01:49.365578 29482 tablet_replica.cc:442] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:49.533550 29483 raft_consensus.cc:482] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:49.533708 29483 raft_consensus.cc:504] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.534935 29483 leader_election.cc:288] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2:35545), 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3:40791)
I20260810 23:01:49.539148 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.535313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:49109 (local address 127.15.56.3:40791)
0810 23:01:49.535432 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.535436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.535768 (+ 332us) server_negotiation.cc:407] Connection header received
0810 23:01:49.535957 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.535961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.536010 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.536072 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:49.536406 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.536895 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.538106 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.538510 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.538605 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.538675 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.538944 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.539022 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:01:49.539149 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.535284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:45023 (local address 127.15.56.2:35545)
0810 23:01:49.535432 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.535436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.535724 (+ 288us) server_negotiation.cc:407] Connection header received
0810 23:01:49.535890 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.535894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.535954 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.536027 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:01:49.536384 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.536900 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.538107 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.538510 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.538604 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.538675 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.538933 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.539013 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:01:49.539179 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.535173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:35545 (local address 127.15.56.1:45023)
0810 23:01:49.535609 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.535624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.535783 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.536066 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.536069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.536088 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:49.536268 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.536274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.537003 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.537007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.537986 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.537996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.538622 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.538626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.538778 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.538837 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.538904 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.539001 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":130,"spinlock_wait_cycles":4992,"thread_start_us":157,"threads_started":1}
I20260810 23:01:49.539170 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.535173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:40791 (local address 127.15.56.1:49109)
0810 23:01:49.535653 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.535692 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.535803 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.536101 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.536104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.536144 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:01:49.536290 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.536296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.537006 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.537010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.537981 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.537991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.538627 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.538630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.538812 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.538844 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.538909 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.539013 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":147,"spinlock_wait_cycles":8704,"thread_start_us":125,"threads_started":1}
I20260810 23:01:49.539809 29116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e45be9ebbd34605b7d28b117eda34cb" candidate_uuid: "e008090f5cfe49c2a4742178498ee4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fe40da1f8f45ffa394c886d2b99122" is_pre_election: true
I20260810 23:01:49.539953 29116 raft_consensus.cc:2454] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e008090f5cfe49c2a4742178498ee4ee in term 0.
I20260810 23:01:49.540133 29357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e45be9ebbd34605b7d28b117eda34cb" candidate_uuid: "e008090f5cfe49c2a4742178498ee4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" is_pre_election: true
I20260810 23:01:49.540261 29357 raft_consensus.cc:2454] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e008090f5cfe49c2a4742178498ee4ee in term 0.
I20260810 23:01:49.540351 28810 leader_election.cc:302] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e008090f5cfe49c2a4742178498ee4ee, f5fe40da1f8f45ffa394c886d2b99122; no voters:
I20260810 23:01:49.540663 29483 raft_consensus.cc:2781] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:49.540735 29483 raft_consensus.cc:482] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:49.540762 29483 raft_consensus.cc:3037] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.541877 29483 raft_consensus.cc:504] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.542250 29483 leader_election.cc:288] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [CANDIDATE]: Term 1 election: Requested vote from peers f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2:35545), 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3:40791)
I20260810 23:01:49.542642 29116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e45be9ebbd34605b7d28b117eda34cb" candidate_uuid: "e008090f5cfe49c2a4742178498ee4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fe40da1f8f45ffa394c886d2b99122"
I20260810 23:01:49.542670 29357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e45be9ebbd34605b7d28b117eda34cb" candidate_uuid: "e008090f5cfe49c2a4742178498ee4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25dd4635913c4fd7bb24e23fe6d32b70"
I20260810 23:01:49.542760 29116 raft_consensus.cc:3037] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.542805 29357 raft_consensus.cc:3037] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.543761 29116 raft_consensus.cc:2454] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e008090f5cfe49c2a4742178498ee4ee in term 1.
I20260810 23:01:49.543783 29357 raft_consensus.cc:2454] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e008090f5cfe49c2a4742178498ee4ee in term 1.
I20260810 23:01:49.544042 28810 leader_election.cc:302] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e008090f5cfe49c2a4742178498ee4ee, f5fe40da1f8f45ffa394c886d2b99122; no voters:
I20260810 23:01:49.544234 29483 raft_consensus.cc:2781] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:49.544457 29483 raft_consensus.cc:686] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [term 1 LEADER]: Becoming Leader. State: Replica: e008090f5cfe49c2a4742178498ee4ee, State: Running, Role: LEADER
I20260810 23:01:49.544703 29483 consensus_queue.cc:237] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.545907 29545 tablet_replica.cc:439] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: TxnStatusTablet state changed. Reason: New leader e008090f5cfe49c2a4742178498ee4ee. Latest consensus state: current_term: 1 leader_uuid: "e008090f5cfe49c2a4742178498ee4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } }
I20260810 23:01:49.546025 29545 tablet_replica.cc:442] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:01:49.546327 29549 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:01:49.546491 29549 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:01:49.547221 28676 catalog_manager.cc:5495] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to e008090f5cfe49c2a4742178498ee4ee (127.15.56.1). New cstate: current_term: 1 leader_uuid: "e008090f5cfe49c2a4742178498ee4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } health_report { overall_health: UNKNOWN } } }
I20260810 23:01:49.562028 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.558404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.15.56.1:41923)
0810 23:01:49.558521 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.558526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.558539 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:01:49.558729 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.558733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.558806 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.558899 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:49.559148 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.559654 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.560535 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.560754 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.560843 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.561179 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:49.561618 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:49.561723 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.561810 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.561874 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:01:49.562067 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.558281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:41923 (local address 127.0.0.1:45742)
0810 23:01:49.558450 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.558467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.558579 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.558928 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.558931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.558941 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:01:49.559048 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.559054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.559802 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.559805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.560416 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.560424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.560854 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.560857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.560972 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.561030 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:01:49.561710 (+ 680us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:01:49.561728 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.561799 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.561940 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 23:01:49.565155 29357 raft_consensus.cc:1261] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Refusing update from remote peer e008090f5cfe49c2a4742178498ee4ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:49.565382 29116 raft_consensus.cc:1261] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Refusing update from remote peer e008090f5cfe49c2a4742178498ee4ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:49.566075 29483 consensus_queue.cc:1035] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:49.566505 29545 consensus_queue.cc:1035] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:49.567627 29482 tablet_replica.cc:439] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70: TxnStatusTablet state changed. Reason: New leader e008090f5cfe49c2a4742178498ee4ee. Latest consensus state: current_term: 1 leader_uuid: "e008090f5cfe49c2a4742178498ee4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } }
I20260810 23:01:49.567782 29482 tablet_replica.cc:442] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:49.570777 29501 tablet_replica.cc:439] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122: TxnStatusTablet state changed. Reason: New leader e008090f5cfe49c2a4742178498ee4ee. Latest consensus state: current_term: 1 leader_uuid: "e008090f5cfe49c2a4742178498ee4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } }
I20260810 23:01:49.570933 29501 tablet_replica.cc:442] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:49.574229 29556 tablet_replica.cc:439] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e008090f5cfe49c2a4742178498ee4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } }
I20260810 23:01:49.574542 29556 tablet_replica.cc:442] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:01:49.574783 29556 tablet_replica.cc:439] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e008090f5cfe49c2a4742178498ee4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } }
I20260810 23:01:49.574870 29556 tablet_replica.cc:442] T 2e45be9ebbd34605b7d28b117eda34cb P e008090f5cfe49c2a4742178498ee4ee: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:01:49.575060 29482 tablet_replica.cc:439] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e008090f5cfe49c2a4742178498ee4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } }
I20260810 23:01:49.575192 29482 tablet_replica.cc:442] T 2e45be9ebbd34605b7d28b117eda34cb P 25dd4635913c4fd7bb24e23fe6d32b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:49.577338 29501 tablet_replica.cc:439] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e008090f5cfe49c2a4742178498ee4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } } }
I20260810 23:01:49.577469 29501 tablet_replica.cc:442] T 2e45be9ebbd34605b7d28b117eda34cb P f5fe40da1f8f45ffa394c886d2b99122: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:01:49.584793 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.579598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.15.56.3:40791)
0810 23:01:49.579738 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.579743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.579781 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:01:49.579873 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.579876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.579945 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.580062 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:49.580304 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.581128 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.582410 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.582755 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.582857 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.583257 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:49.583897 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:49.584003 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.584591 (+ 588us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.584651 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:01:49.598080 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.594070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:39423 (local address 127.15.56.1:41923)
0810 23:01:49.594189 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.594193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.594234 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:01:49.594285 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.594288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.594359 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.594431 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:49.594688 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.595473 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.596738 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.597078 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.597178 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.597557 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:49.597647 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:49.597712 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.597865 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.597939 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:01:49.598625 29561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.593648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:41923 (local address 127.15.56.3:39423)
0810 23:01:49.594049 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.594066 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.594172 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.594447 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.594449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.594457 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:01:49.594590 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.594596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.595583 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.595586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.596616 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.596628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.597186 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.597189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.597333 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.597397 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:01:49.597757 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:01:49.597765 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.598383 (+ 618us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.598471 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":126,"threads_started":1}
I20260810 23:01:49.606930 29116 raft_consensus.cc:1261] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Refusing update from remote peer 25dd4635913c4fd7bb24e23fe6d32b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:49.606976 28922 raft_consensus.cc:1261] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Refusing update from remote peer 25dd4635913c4fd7bb24e23fe6d32b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:49.607474 29518 consensus_queue.cc:1035] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:49.607720 29482 consensus_queue.cc:1035] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:49.633051 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.628780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:49183 (local address 127.15.56.3:40791)
0810 23:01:49.628900 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.628904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.629242 (+ 338us) server_negotiation.cc:407] Connection header received
0810 23:01:49.629421 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.629424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.629484 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.629557 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:49.629965 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.630487 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.631688 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.631987 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.632087 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.632552 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:49.632644 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:49.632693 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.632864 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.632928 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:01:49.633486 29568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.628704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:40791 (local address 127.15.56.1:49183)
0810 23:01:49.629150 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:49.629165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:49.629284 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:49.629676 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:49.629679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:49.629688 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:01:49.629819 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.629827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.630591 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.630595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.631557 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:49.631568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.632184 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.632219 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:49.632324 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.632396 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:01:49.632762 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:01:49.632768 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:01:49.633268 (+ 500us) client_negotiation.cc:240] Negotiation successful
0810 23:01:49.633354 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":115,"threads_started":1}
I20260810 23:01:49.660701 28677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:49.660916 28677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:49.663547 28677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a: Sending DeleteTablet for 3 replicas of tablet aee76f9b912c4afb9d6c3ff2f20e707a
I20260810 23:01:49.664374 29327 tablet_service.cc:1514] Processing DeleteTablet for tablet aee76f9b912c4afb9d6c3ff2f20e707a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:49 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:53580
I20260810 23:01:49.664575 29509 tablet_replica.cc:330] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70: stopping tablet replica
I20260810 23:01:49.664531 29108 tablet_service.cc:1514] Processing DeleteTablet for tablet aee76f9b912c4afb9d6c3ff2f20e707a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:49 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:50490
I20260810 23:01:49.664695 29509 raft_consensus.cc:2229] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:49.664737 29508 tablet_replica.cc:330] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122: stopping tablet replica
I20260810 23:01:49.664861 29508 raft_consensus.cc:2229] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:49.664985 29508 raft_consensus.cc:2258] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:49.665012 29509 raft_consensus.cc:2258] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:49.665874 29509 ts_tablet_manager.cc:1905] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:49.666074 29508 ts_tablet_manager.cc:1905] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:49.667929 28888 tablet_service.cc:1514] Processing DeleteTablet for tablet aee76f9b912c4afb9d6c3ff2f20e707a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:49 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45730
I20260810 23:01:49.668133 29510 tablet_replica.cc:330] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee: stopping tablet replica
I20260810 23:01:49.668238 29510 raft_consensus.cc:2229] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:49.668372 29510 raft_consensus.cc:2258] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:49.669390 29510 ts_tablet_manager.cc:1905] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:49.669842 29509 ts_tablet_manager.cc:1918] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:49.669925 29509 log.cc:1196] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/aee76f9b912c4afb9d6c3ff2f20e707a
I20260810 23:01:49.670274 29509 ts_tablet_manager.cc:1939] T aee76f9b912c4afb9d6c3ff2f20e707a P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting consensus metadata
I20260810 23:01:49.671106 28655 catalog_manager.cc:4841] TS 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3:40791): tablet aee76f9b912c4afb9d6c3ff2f20e707a (table test-table [id=b395e351d8f44dd1a985e1bdfa70816a]) successfully deleted
I20260810 23:01:49.672832 29510 ts_tablet_manager.cc:1918] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:49.672927 29510 log.cc:1196] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/wal/wals/aee76f9b912c4afb9d6c3ff2f20e707a
I20260810 23:01:49.673177 29510 ts_tablet_manager.cc:1939] T aee76f9b912c4afb9d6c3ff2f20e707a P e008090f5cfe49c2a4742178498ee4ee: Deleting consensus metadata
I20260810 23:01:49.673979 28645 catalog_manager.cc:4841] TS e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923): tablet aee76f9b912c4afb9d6c3ff2f20e707a (table test-table [id=b395e351d8f44dd1a985e1bdfa70816a]) successfully deleted
I20260810 23:01:49.678100 29508 ts_tablet_manager.cc:1918] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:49.678369 29508 log.cc:1196] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/aee76f9b912c4afb9d6c3ff2f20e707a
I20260810 23:01:49.678715 29508 ts_tablet_manager.cc:1939] T aee76f9b912c4afb9d6c3ff2f20e707a P f5fe40da1f8f45ffa394c886d2b99122: Deleting consensus metadata
I20260810 23:01:49.679692 28645 catalog_manager.cc:4841] TS f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2:35545): tablet aee76f9b912c4afb9d6c3ff2f20e707a (table test-table [id=b395e351d8f44dd1a985e1bdfa70816a]) successfully deleted
Aug 10 23:01:49 dist-test-slave-6bbx krb5kdc[28599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:01:49 dist-test-slave-6bbx krb5kdc[28599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:49 dist-test-slave-6bbx krb5kdc[28599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:01:49.707168 28755 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:49.698836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.15.56.62:43055)
0810 23:01:49.698967 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.698970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.699348 (+ 378us) server_negotiation.cc:407] Connection header received
0810 23:01:49.699451 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.699455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.699518 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.699609 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:49.700553 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.701433 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.702783 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.703136 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.703241 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.707029 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39050: BlockingRecv error: recv got EOF from 127.0.0.1:39050 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 10 23:01:49 dist-test-slave-6bbx krb5kdc[28599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM
I20260810 23:01:49.719890 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.709336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.15.56.62:43055)
0810 23:01:49.709453 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.709458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.709895 (+ 437us) server_negotiation.cc:407] Connection header received
0810 23:01:49.710016 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.710020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.710075 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.710156 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:49.710903 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.711749 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.712893 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.713251 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.713360 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.716298 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:49.716303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:49.716306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:49.716338 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:49.718012 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:49.718548 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:49.718552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:49.718554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:49.718663 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:49.718952 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:49.718955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:49.718957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:49.719182 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:49.719316 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.719555 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.719712 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:01:49.722728 28677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:01:49.723237 28677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:49.729061 29327 tablet_service.cc:1467] Processing CreateTablet for tablet f86e19597318436ca44630ba0ee821bc (DEFAULT_TABLE table=test-table [id=420a0eb9487748cc99089345dd9d86ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.729135 28888 tablet_service.cc:1467] Processing CreateTablet for tablet f86e19597318436ca44630ba0ee821bc (DEFAULT_TABLE table=test-table [id=420a0eb9487748cc99089345dd9d86ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.729434 29327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f86e19597318436ca44630ba0ee821bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.729451 28888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f86e19597318436ca44630ba0ee821bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.730552 29108 tablet_service.cc:1467] Processing CreateTablet for tablet f86e19597318436ca44630ba0ee821bc (DEFAULT_TABLE table=test-table [id=420a0eb9487748cc99089345dd9d86ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.730863 29108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f86e19597318436ca44630ba0ee821bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.732008 29476 tablet_bootstrap.cc:492] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee: Bootstrap starting.
I20260810 23:01:49.732013 29474 tablet_bootstrap.cc:492] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70: Bootstrap starting.
I20260810 23:01:49.732869 29476 tablet_bootstrap.cc:654] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.732947 29474 tablet_bootstrap.cc:654] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.733237 29473 tablet_bootstrap.cc:492] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122: Bootstrap starting.
I20260810 23:01:49.734050 29473 tablet_bootstrap.cc:654] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.734340 29474 tablet_bootstrap.cc:492] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70: No bootstrap required, opened a new log
I20260810 23:01:49.734434 29474 ts_tablet_manager.cc:1397] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:49.734894 29474 raft_consensus.cc:348] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.735019 29474 raft_consensus.cc:374] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.735056 29474 raft_consensus.cc:729] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25dd4635913c4fd7bb24e23fe6d32b70, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.735150 29476 tablet_bootstrap.cc:492] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee: No bootstrap required, opened a new log
I20260810 23:01:49.735172 29473 tablet_bootstrap.cc:492] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122: No bootstrap required, opened a new log
I20260810 23:01:49.735175 29474 consensus_queue.cc:260] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.735244 29476 ts_tablet_manager.cc:1397] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:49.735244 29473 ts_tablet_manager.cc:1397] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:01:49.735476 29474 ts_tablet_manager.cc:1428] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:49.735636 29473 raft_consensus.cc:348] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.735646 29476 raft_consensus.cc:348] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.735776 29476 raft_consensus.cc:374] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.735776 29473 raft_consensus.cc:374] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.735816 29473 raft_consensus.cc:729] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fe40da1f8f45ffa394c886d2b99122, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.735816 29476 raft_consensus.cc:729] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e008090f5cfe49c2a4742178498ee4ee, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.735932 29473 consensus_queue.cc:260] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.735934 29476 consensus_queue.cc:260] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.736207 29476 ts_tablet_manager.cc:1428] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:49.736207 29473 ts_tablet_manager.cc:1428] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:01:49.910778 29501 raft_consensus.cc:482] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:49.910962 29501 raft_consensus.cc:504] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.911525 29501 leader_election.cc:288] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923), 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3:40791)
I20260810 23:01:49.912130 28922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f86e19597318436ca44630ba0ee821bc" candidate_uuid: "f5fe40da1f8f45ffa394c886d2b99122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e008090f5cfe49c2a4742178498ee4ee" is_pre_election: true
I20260810 23:01:49.912130 29357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f86e19597318436ca44630ba0ee821bc" candidate_uuid: "f5fe40da1f8f45ffa394c886d2b99122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" is_pre_election: true
I20260810 23:01:49.912325 29357 raft_consensus.cc:2454] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5fe40da1f8f45ffa394c886d2b99122 in term 0.
I20260810 23:01:49.912326 28922 raft_consensus.cc:2454] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5fe40da1f8f45ffa394c886d2b99122 in term 0.
I20260810 23:01:49.912681 29035 leader_election.cc:302] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e008090f5cfe49c2a4742178498ee4ee, f5fe40da1f8f45ffa394c886d2b99122; no voters:
I20260810 23:01:49.912873 29501 raft_consensus.cc:2781] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:49.912930 29501 raft_consensus.cc:482] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:49.912954 29501 raft_consensus.cc:3037] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.913997 29501 raft_consensus.cc:504] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.914351 29501 leader_election.cc:288] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [CANDIDATE]: Term 1 election: Requested vote from peers e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923), 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3:40791)
I20260810 23:01:49.914731 28922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f86e19597318436ca44630ba0ee821bc" candidate_uuid: "f5fe40da1f8f45ffa394c886d2b99122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e008090f5cfe49c2a4742178498ee4ee"
I20260810 23:01:49.914764 29357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f86e19597318436ca44630ba0ee821bc" candidate_uuid: "f5fe40da1f8f45ffa394c886d2b99122" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25dd4635913c4fd7bb24e23fe6d32b70"
I20260810 23:01:49.914847 28922 raft_consensus.cc:3037] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.914930 29357 raft_consensus.cc:3037] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:49.916055 28922 raft_consensus.cc:2454] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5fe40da1f8f45ffa394c886d2b99122 in term 1.
I20260810 23:01:49.916263 29357 raft_consensus.cc:2454] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5fe40da1f8f45ffa394c886d2b99122 in term 1.
I20260810 23:01:49.916385 29035 leader_election.cc:302] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e008090f5cfe49c2a4742178498ee4ee, f5fe40da1f8f45ffa394c886d2b99122; no voters:
I20260810 23:01:49.916591 29501 raft_consensus.cc:2781] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:49.916774 29501 raft_consensus.cc:686] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 1 LEADER]: Becoming Leader. State: Replica: f5fe40da1f8f45ffa394c886d2b99122, State: Running, Role: LEADER
I20260810 23:01:49.916927 29501 consensus_queue.cc:237] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.918866 28677 catalog_manager.cc:5495] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2). New cstate: current_term: 1 leader_uuid: "f5fe40da1f8f45ffa394c886d2b99122" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } health_report { overall_health: UNKNOWN } } }
I20260810 23:01:49.932220 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:49.929373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50518 (local address 127.15.56.2:35545)
0810 23:01:49.929512 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:49.929516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:49.929531 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:01:49.929625 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:49.929628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:49.929673 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:49.929733 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:49.929927 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.930503 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.931289 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:49.931514 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:49.931593 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:49.931840 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:49.931912 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:49.931965 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:49.932026 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 23:01:49.932075 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:01:49.934118 29357 raft_consensus.cc:1261] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Refusing update from remote peer f5fe40da1f8f45ffa394c886d2b99122: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:49.934160 28922 raft_consensus.cc:1261] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Refusing update from remote peer f5fe40da1f8f45ffa394c886d2b99122: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:49.934532 29592 consensus_queue.cc:1035] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:49.934718 29501 consensus_queue.cc:1035] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:49.942639 28676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:49.942811 28676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:49.944800 28676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a: Sending DeleteTablet for 3 replicas of tablet f86e19597318436ca44630ba0ee821bc
I20260810 23:01:49.945410 29108 tablet_service.cc:1514] Processing DeleteTablet for tablet f86e19597318436ca44630ba0ee821bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:49 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:50490
I20260810 23:01:49.945425 29327 tablet_service.cc:1514] Processing DeleteTablet for tablet f86e19597318436ca44630ba0ee821bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:49 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:53580
I20260810 23:01:49.945606 29508 tablet_replica.cc:330] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122: stopping tablet replica
I20260810 23:01:49.945619 28888 tablet_service.cc:1514] Processing DeleteTablet for tablet f86e19597318436ca44630ba0ee821bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:49 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45730
I20260810 23:01:49.945748 29508 raft_consensus.cc:2229] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:49.945842 29510 tablet_replica.cc:330] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee: stopping tablet replica
I20260810 23:01:49.945953 29508 raft_consensus.cc:2258] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:49.945962 29510 raft_consensus.cc:2229] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:49.946117 29510 raft_consensus.cc:2258] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:49.946223 29509 tablet_replica.cc:330] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70: stopping tablet replica
I20260810 23:01:49.946336 29509 raft_consensus.cc:2229] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:49.946379 28676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:01:49.946475 29509 raft_consensus.cc:2258] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:01:49.946786 28676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:01:49.946853 29508 ts_tablet_manager.cc:1905] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:49.947316 29509 ts_tablet_manager.cc:1905] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:49.948099 29510 ts_tablet_manager.cc:1905] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:49.949620 29508 ts_tablet_manager.cc:1918] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:49.949702 29508 log.cc:1196] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/f86e19597318436ca44630ba0ee821bc
I20260810 23:01:49.949831 29509 ts_tablet_manager.cc:1918] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:49.949899 29509 log.cc:1196] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/f86e19597318436ca44630ba0ee821bc
I20260810 23:01:49.949956 29508 ts_tablet_manager.cc:1939] T f86e19597318436ca44630ba0ee821bc P f5fe40da1f8f45ffa394c886d2b99122: Deleting consensus metadata
I20260810 23:01:49.950096 29510 ts_tablet_manager.cc:1918] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:01:49.950124 29509 ts_tablet_manager.cc:1939] T f86e19597318436ca44630ba0ee821bc P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting consensus metadata
I20260810 23:01:49.950165 29510 log.cc:1196] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-0/wal/wals/f86e19597318436ca44630ba0ee821bc
I20260810 23:01:49.950340 29510 ts_tablet_manager.cc:1939] T f86e19597318436ca44630ba0ee821bc P e008090f5cfe49c2a4742178498ee4ee: Deleting consensus metadata
I20260810 23:01:49.950991 28655 catalog_manager.cc:4841] TS 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3:40791): tablet f86e19597318436ca44630ba0ee821bc (table test-table [id=420a0eb9487748cc99089345dd9d86ba]) successfully deleted
I20260810 23:01:49.951289 28645 catalog_manager.cc:4841] TS e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923): tablet f86e19597318436ca44630ba0ee821bc (table test-table [id=420a0eb9487748cc99089345dd9d86ba]) successfully deleted
I20260810 23:01:49.951452 28645 catalog_manager.cc:4841] TS f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2:35545): tablet f86e19597318436ca44630ba0ee821bc (table test-table [id=420a0eb9487748cc99089345dd9d86ba]) successfully deleted
I20260810 23:01:49.954512 28888 tablet_service.cc:1467] Processing CreateTablet for tablet 3de59cc38a46493f9cde71ebd51dc37a (DEFAULT_TABLE table=test-table [id=6d6e9d0a4d3044f689f5fd44e9f4cbf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.954540 29108 tablet_service.cc:1467] Processing CreateTablet for tablet 3de59cc38a46493f9cde71ebd51dc37a (DEFAULT_TABLE table=test-table [id=6d6e9d0a4d3044f689f5fd44e9f4cbf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.954712 29327 tablet_service.cc:1467] Processing CreateTablet for tablet 3de59cc38a46493f9cde71ebd51dc37a (DEFAULT_TABLE table=test-table [id=6d6e9d0a4d3044f689f5fd44e9f4cbf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:01:49.954842 28888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3de59cc38a46493f9cde71ebd51dc37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.954864 29108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3de59cc38a46493f9cde71ebd51dc37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.954974 29327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3de59cc38a46493f9cde71ebd51dc37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:49.956851 29476 tablet_bootstrap.cc:492] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee: Bootstrap starting.
I20260810 23:01:49.957230 29473 tablet_bootstrap.cc:492] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122: Bootstrap starting.
I20260810 23:01:49.957726 29476 tablet_bootstrap.cc:654] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.957809 29474 tablet_bootstrap.cc:492] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70: Bootstrap starting.
I20260810 23:01:49.958034 29473 tablet_bootstrap.cc:654] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.958654 29474 tablet_bootstrap.cc:654] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:49.959160 29473 tablet_bootstrap.cc:492] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122: No bootstrap required, opened a new log
I20260810 23:01:49.959266 29476 tablet_bootstrap.cc:492] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee: No bootstrap required, opened a new log
I20260810 23:01:49.959275 29473 ts_tablet_manager.cc:1397] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:01:49.959347 29476 ts_tablet_manager.cc:1397] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 23:01:49.959725 29474 tablet_bootstrap.cc:492] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70: No bootstrap required, opened a new log
I20260810 23:01:49.959733 29476 raft_consensus.cc:348] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.959719 29473 raft_consensus.cc:348] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.959815 29474 ts_tablet_manager.cc:1397] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:49.959841 29473 raft_consensus.cc:374] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.959841 29476 raft_consensus.cc:374] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.959878 29476 raft_consensus.cc:729] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e008090f5cfe49c2a4742178498ee4ee, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.959878 29473 raft_consensus.cc:729] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fe40da1f8f45ffa394c886d2b99122, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.960005 29476 consensus_queue.cc:260] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.960005 29473 consensus_queue.cc:260] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.960281 29476 ts_tablet_manager.cc:1428] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:49.960278 29473 ts_tablet_manager.cc:1428] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:01:49.960253 29474 raft_consensus.cc:348] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.960357 29474 raft_consensus.cc:374] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:49.960393 29474 raft_consensus.cc:729] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25dd4635913c4fd7bb24e23fe6d32b70, State: Initialized, Role: FOLLOWER
I20260810 23:01:49.960521 29474 consensus_queue.cc:260] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:49.961771 29474 ts_tablet_manager.cc:1428] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:01:50.220456 29482 raft_consensus.cc:482] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:01:50.220624 29482 raft_consensus.cc:504] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:50.221063 29482 leader_election.cc:288] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923), f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2:35545)
I20260810 23:01:50.221601 28922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3de59cc38a46493f9cde71ebd51dc37a" candidate_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e008090f5cfe49c2a4742178498ee4ee" is_pre_election: true
I20260810 23:01:50.221638 29116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3de59cc38a46493f9cde71ebd51dc37a" candidate_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fe40da1f8f45ffa394c886d2b99122" is_pre_election: true
I20260810 23:01:50.221783 29116 raft_consensus.cc:2454] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25dd4635913c4fd7bb24e23fe6d32b70 in term 0.
I20260810 23:01:50.221805 28922 raft_consensus.cc:2454] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25dd4635913c4fd7bb24e23fe6d32b70 in term 0.
I20260810 23:01:50.222142 29246 leader_election.cc:302] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25dd4635913c4fd7bb24e23fe6d32b70, f5fe40da1f8f45ffa394c886d2b99122; no voters:
I20260810 23:01:50.222335 29482 raft_consensus.cc:2781] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:01:50.222398 29482 raft_consensus.cc:482] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:01:50.222424 29482 raft_consensus.cc:3037] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:50.223448 29482 raft_consensus.cc:504] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:50.223824 29482 leader_election.cc:288] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [CANDIDATE]: Term 1 election: Requested vote from peers e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923), f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2:35545)
I20260810 23:01:50.224291 28922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3de59cc38a46493f9cde71ebd51dc37a" candidate_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e008090f5cfe49c2a4742178498ee4ee"
I20260810 23:01:50.224337 29116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3de59cc38a46493f9cde71ebd51dc37a" candidate_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5fe40da1f8f45ffa394c886d2b99122"
I20260810 23:01:50.224402 28922 raft_consensus.cc:3037] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:50.224470 29116 raft_consensus.cc:3037] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:50.225339 28922 raft_consensus.cc:2454] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25dd4635913c4fd7bb24e23fe6d32b70 in term 1.
I20260810 23:01:50.225443 29116 raft_consensus.cc:2454] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25dd4635913c4fd7bb24e23fe6d32b70 in term 1.
I20260810 23:01:50.225664 29246 leader_election.cc:302] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25dd4635913c4fd7bb24e23fe6d32b70, e008090f5cfe49c2a4742178498ee4ee; no voters:
I20260810 23:01:50.225994 29482 raft_consensus.cc:2781] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:50.226179 29482 raft_consensus.cc:686] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 LEADER]: Becoming Leader. State: Replica: 25dd4635913c4fd7bb24e23fe6d32b70, State: Running, Role: LEADER
I20260810 23:01:50.226356 29482 consensus_queue.cc:237] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } }
I20260810 23:01:50.228302 28677 catalog_manager.cc:5495] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3). New cstate: current_term: 1 leader_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25dd4635913c4fd7bb24e23fe6d32b70" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40791 } health_report { overall_health: HEALTHY } } }
I20260810 23:01:50.265843 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:50.262813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.15.56.3:40791)
0810 23:01:50.263095 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:50.263098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:50.263111 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:01:50.263158 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:50.263160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:50.263206 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:50.263284 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:50.263589 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:50.264148 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:50.264984 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:50.265201 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:50.265270 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:50.265467 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:50.265539 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:50.265587 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:50.265657 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 23:01:50.265702 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":121,"threads_started":1}
I20260810 23:01:50.274044 29116 raft_consensus.cc:1261] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Refusing update from remote peer 25dd4635913c4fd7bb24e23fe6d32b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:50.274058 28922 raft_consensus.cc:1261] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Refusing update from remote peer 25dd4635913c4fd7bb24e23fe6d32b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:50.274507 29482 consensus_queue.cc:1035] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e008090f5cfe49c2a4742178498ee4ee" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:50.274683 29606 consensus_queue.cc:1035] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fe40da1f8f45ffa394c886d2b99122" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:50.310724 28677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:50.310889 28677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:01:50.312958 28677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdae613b93a24f13a69b59b75f18988a: Sending DeleteTablet for 3 replicas of tablet 3de59cc38a46493f9cde71ebd51dc37a
I20260810 23:01:50.313793 28888 tablet_service.cc:1514] Processing DeleteTablet for tablet 3de59cc38a46493f9cde71ebd51dc37a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:50 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45730
I20260810 23:01:50.313915 29108 tablet_service.cc:1514] Processing DeleteTablet for tablet 3de59cc38a46493f9cde71ebd51dc37a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:50 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:50490
I20260810 23:01:50.314105 29508 tablet_replica.cc:330] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122: stopping tablet replica
I20260810 23:01:50.314165 29327 tablet_service.cc:1514] Processing DeleteTablet for tablet 3de59cc38a46493f9cde71ebd51dc37a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:01:50 UTC) from {username='oryx', principal='oryx/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:53580
I20260810 23:01:50.314235 29508 raft_consensus.cc:2229] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:50.314368 29508 raft_consensus.cc:2258] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:50.314363 29509 tablet_replica.cc:330] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70: stopping tablet replica
I20260810 23:01:50.314513 29509 raft_consensus.cc:2229] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:01:50.314602 29510 tablet_replica.cc:330] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee: stopping tablet replica
I20260810 23:01:50.314750 29509 raft_consensus.cc:2258] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:50.314776 29510 raft_consensus.cc:2229] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:01:50.314944 29510 raft_consensus.cc:2258] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:01:50.315157 29508 ts_tablet_manager.cc:1905] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:50.315541 29509 ts_tablet_manager.cc:1905] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:50.316747 29510 ts_tablet_manager.cc:1905] T 3de59cc38a46493f9cde71ebd51dc37a P e008090f5cfe49c2a4742178498ee4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:01:50.317782 29508 ts_tablet_manager.cc:1918] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:50.317862 29508 log.cc:1196] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/3de59cc38a46493f9cde71ebd51dc37a
I20260810 23:01:50.318070 29508 ts_tablet_manager.cc:1939] T 3de59cc38a46493f9cde71ebd51dc37a P f5fe40da1f8f45ffa394c886d2b99122: Deleting consensus metadata
I20260810 23:01:50.318186 29509 ts_tablet_manager.cc:1918] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:01:50.318279 29509 log.cc:1196] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/3de59cc38a46493f9cde71ebd51dc37a
I20260810 23:01:50.318550 29509 ts_tablet_manager.cc:1939] T 3de59cc38a46493f9cde71ebd51dc37a P 25dd4635913c4fd7bb24e23fe6d32b70: Deleting consensus metadata
I20260810 23:01:50.318984 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 28787
I20260810 23:01:50.319087 28645 catalog_manager.cc:4841] TS f5fe40da1f8f45ffa394c886d2b99122 (127.15.56.2:35545): tablet 3de59cc38a46493f9cde71ebd51dc37a (table test-table [id=6d6e9d0a4d3044f689f5fd44e9f4cbf4]) successfully deleted
I20260810 23:01:50.325760 28655 catalog_manager.cc:4841] TS 25dd4635913c4fd7bb24e23fe6d32b70 (127.15.56.3:40791): tablet 3de59cc38a46493f9cde71ebd51dc37a (table test-table [id=6d6e9d0a4d3044f689f5fd44e9f4cbf4]) successfully deleted
I20260810 23:01:50.337894 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 29012
W20260810 23:01:50.337905 28645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:41923 (error 108)
I20260810 23:01:50.345223 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:50.344816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:41923 (local address 127.0.0.1:45748)
0810 23:01:50.345051 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:50.345125 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:41923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
W20260810 23:01:50.345404 28645 catalog_manager.cc:4584] TS e008090f5cfe49c2a4742178498ee4ee (127.15.56.1:41923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3de59cc38a46493f9cde71ebd51dc37a: Network error: Client connection negotiation failed: client connection to 127.15.56.1:41923: connect: Connection refused (error 111)
I20260810 23:01:50.351531 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 29225
I20260810 23:01:50.367430 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 28620
2026-08-10T23:01:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3601 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:50 dist-test-slave-6bbx krb5kdc[29631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:50 dist-test-slave-6bbx krb5kdc[29631](info): set up 2 sockets
Aug 10 23:01:50 dist-test-slave-6bbx krb5kdc[29631](info): commencing operation
krb5kdc: starting...
W20260810 23:01:51.536000 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:51 dist-test-slave-6bbx krb5kdc[29631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:51.664938 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:38615
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:36451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:51.780750 29652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:51.781050 29652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:51.781106 29652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:51.785152 29652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:51.785230 29652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:51.785259 29652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:51.785279 29652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:51.785298 29652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:51.790859 29652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:36451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:51.792069 29652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:51.793157 29652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:51.801854 29665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:51.801856 29659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:51.801856 29660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:51.802016 29652 server_base.cc:1034] running on GCE node
I20260810 23:01:51.802878 29652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:51.804117 29652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:51.805284 29652 hybrid_clock.cc:648] HybridClock initialized: now 1786402911805260 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T23:01:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1450 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:51 dist-test-slave-6bbx krb5kdc[29673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:51 dist-test-slave-6bbx krb5kdc[29673](info): set up 2 sockets
Aug 10 23:01:51 dist-test-slave-6bbx krb5kdc[29673](info): commencing operation
krb5kdc: starting...
W20260810 23:01:52.984863 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:01:53 dist-test-slave-6bbx krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:01:53.118042 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44491
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:44491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:53.235394 29696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:53.235699 29696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:53.235755 29696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:53.239737 29696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:53.239805 29696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:53.239830 29696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:53.239845 29696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:53.239859 29696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:53.245371 29696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:44491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:53.246510 29696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:53.247514 29696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:53.256256 29706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:53.256286 29708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:53.256273 29705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:53.256580 29696 server_base.cc:1034] running on GCE node
I20260810 23:01:53.257220 29696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:53.258455 29696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:53.259631 29696 hybrid_clock.cc:648] HybridClock initialized: now 1786402913259611 us; error 35 us; skew 500 ppm
Aug 10 23:01:53 dist-test-slave-6bbx krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402913, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:53.262748 29696 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:53.263796 29696 webserver.cc:460] Webserver started at http://127.15.56.62:46821/ using document root <none> and password file <none>
I20260810 23:01:53.264416 29696 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:53.264487 29696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:53.264712 29696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:53.266940 29696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "1c8f205e472b468faf3000b0afff2ecf"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "5c8e5e5940a21cb6f0c201f3192ba361"
server_key_iv: "bf7f4a95839f48036d01d81d1da4eb20"
server_key_version: "encryptionkey@0"
I20260810 23:01:53.267534 29696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "1c8f205e472b468faf3000b0afff2ecf"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "5c8e5e5940a21cb6f0c201f3192ba361"
server_key_iv: "bf7f4a95839f48036d01d81d1da4eb20"
server_key_version: "encryptionkey@0"
I20260810 23:01:53.271484 29696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 23:01:53.274112 29717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:53.275538 29696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:01:53.275733 29696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "1c8f205e472b468faf3000b0afff2ecf"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "5c8e5e5940a21cb6f0c201f3192ba361"
server_key_iv: "bf7f4a95839f48036d01d81d1da4eb20"
server_key_version: "encryptionkey@0"
I20260810 23:01:53.275856 29696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:53.295938 29696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:53.299448 29696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:53.299655 29696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:53.307284 29696 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:44491
I20260810 23:01:53.307375 29797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:44491 every 8 connection(s)
I20260810 23:01:53.308391 29696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:53.311553 29799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:53.315200 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 29696
I20260810 23:01:53.315340 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:01:53.315609 15584 external_mini_cluster.cc:1428] Setting key 76a474736a88369cdae82bd93301894b
I20260810 23:01:53.318318 29799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf: Bootstrap starting.
I20260810 23:01:53.321382 29799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:53.322387 29799 log.cc:824] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:53.325484 29799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf: No bootstrap required, opened a new log
Aug 10 23:01:53 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:53.329870 29799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } }
I20260810 23:01:53.330199 29799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:01:53.330281 29799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8f205e472b468faf3000b0afff2ecf, State: Initialized, Role: FOLLOWER
I20260810 23:01:53.330850 29799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } }
I20260810 23:01:53.331009 29799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:53.331096 29799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:53.331284 29799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:53.332366 29799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } }
I20260810 23:01:53.332756 29799 leader_election.cc:302] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c8f205e472b468faf3000b0afff2ecf; no voters:
I20260810 23:01:53.333101 29799 leader_election.cc:288] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:01:53.333201 29808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:53.333367 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:53.317305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.15.56.62:44491)
0810 23:01:53.317802 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:53.317823 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:53.317872 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:01:53.318747 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:53.318776 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:53.319283 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:53.319753 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:53.320466 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:53.321965 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:53.323023 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:53.323690 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:53.323870 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:53.327197 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:53.327205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:53.327226 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:53.327263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:53.329977 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:53.330534 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:53.330541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:53.330547 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:53.330704 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:53.331071 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:53.331075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:53.331076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:53.331567 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:53.331810 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:53.332123 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 23:01:53.332453 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":117,"threads_started":1}
I20260810 23:01:53.333421 29808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 1 LEADER]: Becoming Leader. State: Replica: 1c8f205e472b468faf3000b0afff2ecf, State: Running, Role: LEADER
I20260810 23:01:53.333822 29808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } }
I20260810 23:01:53.334535 29799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:53.335815 29811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c8f205e472b468faf3000b0afff2ecf. Latest consensus state: current_term: 1 leader_uuid: "1c8f205e472b468faf3000b0afff2ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } } }
I20260810 23:01:53.336019 29811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:53.335592 29809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c8f205e472b468faf3000b0afff2ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } } }
I20260810 23:01:53.336436 29809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [sys.catalog]: This master's current role is: LEADER
W20260810 23:01:53.339977 29834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:01:53.340054 29834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:01:53.340137 29826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:53.340848 29826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:53.346998 29826 catalog_manager.cc:1347] Generated new cluster ID: 1e7a4f9a6af34d4185e957f8715609ba
I20260810 23:01:53.347064 29826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:53.361088 29826 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:53.362095 29826 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:53.381422 29826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf: Generated new TSK 0
I20260810 23:01:53.382236 29826 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:01:53.413954 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44491
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:53.535429 29841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:53.535809 29841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:53.535867 29841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:53.540153 29841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:53.540236 29841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:53.540300 29841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:53.545912 29841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:53.547086 29841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:53.548198 29841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:53.558771 29859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:53.558771 29852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:53.558993 29841 server_base.cc:1034] running on GCE node
W20260810 23:01:53.558795 29851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:53.559589 29841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:53.560335 29841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:53.561511 29841 hybrid_clock.cc:648] HybridClock initialized: now 1786402913561467 us; error 55 us; skew 500 ppm
Aug 10 23:01:53 dist-test-slave-6bbx krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402913, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:53.564436 29841 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:01:53.565333 29841 webserver.cc:460] Webserver started at http://127.15.56.1:32863/ using document root <none> and password file <none>
I20260810 23:01:53.565869 29841 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:53.565928 29841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:53.566102 29841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:53.568207 29841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "327a57a877ed4dbcadc69b7c0157ba99"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "da177a9ed35f4edc6ac45a02dc61e2f0"
server_key_iv: "47a5eb776b5361e872a6836bcd438f78"
server_key_version: "encryptionkey@0"
I20260810 23:01:53.568719 29841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "327a57a877ed4dbcadc69b7c0157ba99"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "da177a9ed35f4edc6ac45a02dc61e2f0"
server_key_iv: "47a5eb776b5361e872a6836bcd438f78"
server_key_version: "encryptionkey@0"
I20260810 23:01:53.572507 29841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:01:53.574950 29870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:53.576328 29841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:01:53.576437 29841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "327a57a877ed4dbcadc69b7c0157ba99"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "da177a9ed35f4edc6ac45a02dc61e2f0"
server_key_iv: "47a5eb776b5361e872a6836bcd438f78"
server_key_version: "encryptionkey@0"
I20260810 23:01:53.576534 29841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:53.594097 29841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:53.600471 29841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:53.600703 29841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:53.601428 29841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:53.602509 29841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:53.602561 29841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:53.602607 29841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:53.602634 29841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:53.611059 29841 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:37235
I20260810 23:01:53.611099 30069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:37235 every 8 connection(s)
I20260810 23:01:53.612064 29841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:53.621559 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 29841
Aug 10 23:01:53 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402913, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:53.621690 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:01:53.621991 15584 external_mini_cluster.cc:1428] Setting key f03d50b4f97564f640ee7028f64bc8da
I20260810 23:01:53.625877 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:53.613882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:37039 (local address 127.15.56.62:44491)
0810 23:01:53.614004 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:53.614008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:53.614883 (+ 875us) server_negotiation.cc:407] Connection header received
0810 23:01:53.616270 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:53.616274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:53.616345 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:53.616419 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:53.617674 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:53.618210 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:53.619367 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:53.619595 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:53.619711 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:53.622285 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:53.622288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:53.622291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:53.622330 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:53.623756 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:53.624288 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:53.624291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:53.624293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:53.624395 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:53.624691 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:53.624694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:53.624695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:53.624906 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:53.624998 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:53.625548 (+ 550us) server_negotiation.cc:299] Negotiation successful
0810 23:01:53.625688 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:01:53.626818 30072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:53.614260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:37039)
0810 23:01:53.614727 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:53.614772 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:53.616052 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:53.616615 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:53.616622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:53.617006 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:53.617518 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:53.617533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:53.618332 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:53.618335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:53.619238 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:53.619247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:53.619772 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:53.619776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:53.619947 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:53.620515 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:53.620532 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:53.622090 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:53.623884 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:53.623891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:53.623906 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:53.624179 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:53.624478 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:53.624480 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:53.624482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:53.624618 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:53.624996 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:53.625001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:53.625286 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 23:01:53.625484 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 23:01:53.625847 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":129,"threads_started":1}
I20260810 23:01:53.628036 30070 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44491
I20260810 23:01:53.628334 30070 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:53.628957 30070 heartbeater.cc:511] Master 127.15.56.62:44491 requested a full tablet report, sending...
I20260810 23:01:53.630576 29742 ts_manager.cc:194] Registered new tserver with Master: 327a57a877ed4dbcadc69b7c0157ba99 (127.15.56.1:37235)
I20260810 23:01:53.631775 29742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:37039
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:01:53.655588 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44491
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:53.772425 30075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:53.772716 30075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:53.772768 30075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:53.776990 30075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:53.777056 30075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:53.777124 30075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:53.782687 30075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:53.783851 30075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:53.784960 30075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:53.795189 30085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:53.795183 30090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:53.795189 30086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:53.795379 30075 server_base.cc:1034] running on GCE node
I20260810 23:01:53.796031 30075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:53.796765 30075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:53.797986 30075 hybrid_clock.cc:648] HybridClock initialized: now 1786402913797960 us; error 44 us; skew 500 ppm
Aug 10 23:01:53 dist-test-slave-6bbx krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402913, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:53.801239 30075 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:01:53.802368 30075 webserver.cc:460] Webserver started at http://127.15.56.2:38387/ using document root <none> and password file <none>
I20260810 23:01:53.802989 30075 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:53.803061 30075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:53.803254 30075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:53.805549 30075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "cf74055436c943678907f51e395fa115"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "f79bde24d4d844ac55e2fbe0ff336ea5"
server_key_iv: "c59c3ca47baf61da6f625ca7cecce5c8"
server_key_version: "encryptionkey@0"
I20260810 23:01:53.806103 30075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "cf74055436c943678907f51e395fa115"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "f79bde24d4d844ac55e2fbe0ff336ea5"
server_key_iv: "c59c3ca47baf61da6f625ca7cecce5c8"
server_key_version: "encryptionkey@0"
I20260810 23:01:53.810029 30075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 23:01:53.812669 30101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:53.813947 30075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:01:53.814054 30075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "cf74055436c943678907f51e395fa115"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "f79bde24d4d844ac55e2fbe0ff336ea5"
server_key_iv: "c59c3ca47baf61da6f625ca7cecce5c8"
server_key_version: "encryptionkey@0"
I20260810 23:01:53.814169 30075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:53.831568 30075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:53.835273 30075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:53.835513 30075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:53.836228 30075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:53.837168 30075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:53.837232 30075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:53.837286 30075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:53.837308 30075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:53.846323 30075 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:41081
I20260810 23:01:53.846367 30292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:41081 every 8 connection(s)
I20260810 23:01:53.847265 30075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:53.853596 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 30075
I20260810 23:01:53.853720 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:01:53.854015 15584 external_mini_cluster.cc:1428] Setting key ddb1f40efef26e867fc8d1cad519448f
Aug 10 23:01:53 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402913, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:53.862059 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:53.849223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:39217 (local address 127.15.56.62:44491)
0810 23:01:53.849368 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:53.849372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:53.850294 (+ 922us) server_negotiation.cc:407] Connection header received
0810 23:01:53.851659 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:53.851690 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:53.851755 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:53.851828 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:53.853152 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:53.853838 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:53.855318 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:53.855614 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:53.855754 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:53.858447 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:53.858451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:53.858453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:53.858479 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:53.859879 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:53.860411 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:53.860414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:53.860415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:53.860495 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:53.860893 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:53.860896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:53.860898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:53.861090 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:53.861169 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:53.861745 (+ 576us) server_negotiation.cc:299] Negotiation successful
0810 23:01:53.861888 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:01:53.863037 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:53.849656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:39217)
0810 23:01:53.850131 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:53.850178 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:53.851429 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:53.852043 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:53.852051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:53.852433 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:53.853000 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:53.853015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:53.853969 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:53.853973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:53.855178 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:53.855190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:53.855813 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:53.855816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:53.855972 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:53.856605 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:53.856624 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:53.858258 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:53.859996 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:53.860003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:53.860024 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:53.860302 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:53.860589 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:53.860592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:53.860594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:53.860795 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:53.861204 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:53.861211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:53.861500 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 23:01:53.861684 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 23:01:53.862062 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":127,"threads_started":1}
I20260810 23:01:53.864552 30293 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44491
I20260810 23:01:53.864848 30293 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:01:53.865482 30293 heartbeater.cc:511] Master 127.15.56.62:44491 requested a full tablet report, sending...
I20260810 23:01:53.866935 29742 ts_manager.cc:194] Registered new tserver with Master: cf74055436c943678907f51e395fa115 (127.15.56.2:41081)
I20260810 23:01:53.867563 29742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:39217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:01:53.888669 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44491
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:54.005667 30298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:54.005981 30298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:54.006048 30298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:54.010144 30298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:54.010213 30298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:54.010296 30298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:54.015806 30298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:54.017038 30298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:54.018231 30298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:54.028659 30315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:54.028661 30313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:54.028661 30308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:54.028957 30298 server_base.cc:1034] running on GCE node
I20260810 23:01:54.029608 30298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:54.030337 30298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:54.031534 30298 hybrid_clock.cc:648] HybridClock initialized: now 1786402914031517 us; error 32 us; skew 500 ppm
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402914, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:54.034677 30298 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:01:54.035787 30298 webserver.cc:460] Webserver started at http://127.15.56.3:35237/ using document root <none> and password file <none>
I20260810 23:01:54.036432 30298 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:54.036511 30298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:54.036737 30298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:54.038971 30298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "dd48f69f13544947a146705758fe4bef"
format_stamp: "Formatted at 2026-08-10 23:01:54 on dist-test-slave-6bbx"
server_key: "4dc567ae6274d0d0801de0963930fcc1"
server_key_iv: "db7f99631026ca140e212c265678890b"
server_key_version: "encryptionkey@0"
I20260810 23:01:54.039594 30298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "dd48f69f13544947a146705758fe4bef"
format_stamp: "Formatted at 2026-08-10 23:01:54 on dist-test-slave-6bbx"
server_key: "4dc567ae6274d0d0801de0963930fcc1"
server_key_iv: "db7f99631026ca140e212c265678890b"
server_key_version: "encryptionkey@0"
I20260810 23:01:54.043773 30298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:01:54.046597 30322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:54.048172 30298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:54.048327 30298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "dd48f69f13544947a146705758fe4bef"
format_stamp: "Formatted at 2026-08-10 23:01:54 on dist-test-slave-6bbx"
server_key: "4dc567ae6274d0d0801de0963930fcc1"
server_key_iv: "db7f99631026ca140e212c265678890b"
server_key_version: "encryptionkey@0"
I20260810 23:01:54.048452 30298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:54.065662 30298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:54.069036 30298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:54.069237 30298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:54.069878 30298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:54.070760 30298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:54.070813 30298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:54.070870 30298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:54.070899 30298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:54.080284 30298 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:35765
I20260810 23:01:54.080309 30507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:35765 every 8 connection(s)
I20260810 23:01:54.081339 30298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:01:54.086292 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 30298
I20260810 23:01:54.086439 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:01:54.086714 15584 external_mini_cluster.cc:1428] Setting key 67ef4d84485efafaaa37cabc131ad6eb
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402914, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:01:54.095489 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:54.083261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:47095 (local address 127.15.56.62:44491)
0810 23:01:54.083390 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:54.083393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:54.084424 (+ 1031us) server_negotiation.cc:407] Connection header received
0810 23:01:54.085785 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:54.085789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:54.085875 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:54.085949 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:54.087449 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.088126 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.089297 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.089526 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.089615 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.092090 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:54.092094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:54.092097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:54.092125 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:54.093409 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:54.093887 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:54.093891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:54.093893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:54.093972 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:54.094289 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:54.094292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:54.094294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:54.094486 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:54.094574 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:54.095157 (+ 583us) server_negotiation.cc:299] Negotiation successful
0810 23:01:54.095310 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:01:54.096647 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:54.083714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:47095)
0810 23:01:54.084235 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:54.084283 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:54.085550 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:54.086172 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:54.086180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:54.086631 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:54.087280 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.087314 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.088256 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.088259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.089174 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.089184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.089672 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.089676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.089848 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.090415 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:54.090434 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:54.091929 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:54.093517 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:54.093523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:54.093541 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:54.093794 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:54.094052 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:54.094055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:54.094057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:54.094201 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:54.094586 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:54.094593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:54.094866 (+ 273us) client_negotiation.cc:769] Sending connection context
0810 23:01:54.095081 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 23:01:54.095491 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260810 23:01:54.097841 30510 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44491
I20260810 23:01:54.098100 30510 heartbeater.cc:464] Registering TS with master...
I20260810 23:01:54.098642 30510 heartbeater.cc:511] Master 127.15.56.62:44491 requested a full tablet report, sending...
I20260810 23:01:54.099764 29742 ts_manager.cc:194] Registered new tserver with Master: dd48f69f13544947a146705758fe4bef (127.15.56.3:35765)
I20260810 23:01:54.100494 29742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:47095
I20260810 23:01:54.100684 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab.
Entry for principal oryx/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab.
WARNING: no policy specified for oryx/127.15.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab.
Entry for principal oryx/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab.
WARNING: no policy specified for oryx/127.15.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab.
Entry for principal oryx/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab.
I20260810 23:01:54.184554 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 29841
I20260810 23:01:54.200531 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 30075
I20260810 23:01:54.209925 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 30298
I20260810 23:01:54.236430 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 29696
I20260810 23:01:54.248896 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44491
--webserver_interface=127.15.56.62
--webserver_port=46821
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:44491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:54.364849 30524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:54.365144 30524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:54.365185 30524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:54.369325 30524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:54.369400 30524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:54.369419 30524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:54.369434 30524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:54.369448 30524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:54.374955 30524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:44491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=46821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:54.376127 30524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:54.377213 30524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:54.386284 30533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:54.386358 30524 server_base.cc:1034] running on GCE node
W20260810 23:01:54.386289 30531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:54.386384 30530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:54.387248 30524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:54.388499 30524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:54.389672 30524 hybrid_clock.cc:648] HybridClock initialized: now 1786402914389653 us; error 30 us; skew 500 ppm
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402914, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:54.392634 30524 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:01:54.393675 30524 webserver.cc:460] Webserver started at http://127.15.56.62:46821/ using document root <none> and password file <none>
I20260810 23:01:54.394281 30524 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:54.394377 30524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:54.398586 30524 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 23:01:54.400925 30541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:54.402382 30524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:01:54.402523 30524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "1c8f205e472b468faf3000b0afff2ecf"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "5c8e5e5940a21cb6f0c201f3192ba361"
server_key_iv: "bf7f4a95839f48036d01d81d1da4eb20"
server_key_version: "encryptionkey@0"
I20260810 23:01:54.403112 30524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:54.418924 30524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:54.422353 30524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:54.422554 30524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:54.429828 30524 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:44491
I20260810 23:01:54.429847 30629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:44491 every 8 connection(s)
I20260810 23:01:54.430961 30524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:54.435256 30631 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:01:54.436236 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 30524
I20260810 23:01:54.436646 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:37235
--local_ip_for_outbound_sockets=127.15.56.1
--tserver_master_addrs=127.15.56.62:44491
--webserver_port=32863
--webserver_interface=127.15.56.1
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.56.1
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:01:54.437054 30631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf: Bootstrap starting.
I20260810 23:01:54.454100 30631 log.cc:824] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:54.461226 30631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:01:54.461604 30631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf: Bootstrap complete.
I20260810 23:01:54.465788 30631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } }
I20260810 23:01:54.466187 30631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8f205e472b468faf3000b0afff2ecf, State: Initialized, Role: FOLLOWER
I20260810 23:01:54.466694 30631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } }
I20260810 23:01:54.466820 30631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:01:54.466890 30631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:01:54.466987 30631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:01:54.468932 30631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } }
I20260810 23:01:54.469244 30631 leader_election.cc:302] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c8f205e472b468faf3000b0afff2ecf; no voters:
I20260810 23:01:54.469568 30631 leader_election.cc:288] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:01:54.469669 30640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:01:54.469878 30640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [term 2 LEADER]: Becoming Leader. State: Replica: 1c8f205e472b468faf3000b0afff2ecf, State: Running, Role: LEADER
I20260810 23:01:54.470170 30640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } }
I20260810 23:01:54.470819 30631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:54.471550 30641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1c8f205e472b468faf3000b0afff2ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } } }
I20260810 23:01:54.471699 30641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:54.472074 30645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c8f205e472b468faf3000b0afff2ecf. Latest consensus state: current_term: 2 leader_uuid: "1c8f205e472b468faf3000b0afff2ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c8f205e472b468faf3000b0afff2ecf" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44491 } } }
I20260810 23:01:54.472218 30645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:54.472544 30657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:54.475828 30657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:54.476534 30657 catalog_manager.cc:1259] Loaded cluster ID: 1e7a4f9a6af34d4185e957f8715609ba
I20260810 23:01:54.476603 30657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:54.479085 30657 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:54.479902 30657 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1c8f205e472b468faf3000b0afff2ecf: Loaded TSK: 0
I20260810 23:01:54.481194 30657 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:01:54.557633 30635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:54.557972 30635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:54.558041 30635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:54.562341 30635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:54.562420 30635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:54.562506 30635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:01:54.568045 30635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.1.keytab
--principal=oryx/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:37235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=32863
--tserver_master_addrs=127.15.56.62:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:54.569139 30635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:54.570185 30635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:54.580340 30682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:54.580466 30685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:54.580341 30680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:54.580608 30635 server_base.cc:1034] running on GCE node
I20260810 23:01:54.581177 30635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:54.581852 30635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:54.583036 30635 hybrid_clock.cc:648] HybridClock initialized: now 1786402914583006 us; error 51 us; skew 500 ppm
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402914, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:54.586383 30635 init.cc:377] Logged in from keytab as oryx/127.15.56.1@KRBTEST.COM (short username oryx)
I20260810 23:01:54.587370 30635 webserver.cc:460] Webserver started at http://127.15.56.1:32863/ using document root <none> and password file <none>
I20260810 23:01:54.587996 30635 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:54.588078 30635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:54.592312 30635 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 23:01:54.594609 30698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:54.596235 30635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:54.596376 30635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "327a57a877ed4dbcadc69b7c0157ba99"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "da177a9ed35f4edc6ac45a02dc61e2f0"
server_key_iv: "47a5eb776b5361e872a6836bcd438f78"
server_key_version: "encryptionkey@0"
I20260810 23:01:54.596969 30635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:54.613772 30635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:54.617229 30635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:54.617424 30635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:54.618055 30635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:54.618929 30635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:54.618980 30635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:54.619037 30635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:54.619062 30635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:54.628190 30635 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:37235
I20260810 23:01:54.628232 30872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:37235 every 8 connection(s)
I20260810 23:01:54.629355 30635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:01:54.635011 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 30635
I20260810 23:01:54.635483 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:41081
--local_ip_for_outbound_sockets=127.15.56.2
--tserver_master_addrs=127.15.56.62:44491
--webserver_port=38387
--webserver_interface=127.15.56.2
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.56.2
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:01:54.643249 30879 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:54.631466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:58955 (local address 127.15.56.62:44491)
0810 23:01:54.631981 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:54.632000 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:54.632166 (+ 166us) server_negotiation.cc:407] Connection header received
0810 23:01:54.633655 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:54.633667 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:54.633962 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:54.634233 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:54.635392 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.636457 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.637996 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.638506 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.638640 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.642464 (+ 3824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:58955: BlockingRecv error: recv got EOF from 127.15.56.1:58955 (error 108)
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":161,"threads_started":1}
I20260810 23:01:54.643258 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:54.631519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:58955)
0810 23:01:54.632011 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:54.632064 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:54.633277 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:54.634302 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:54.634308 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:54.634678 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:54.635215 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.635232 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.636602 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.636606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.637844 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.637857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.638679 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.638683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.638867 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.639494 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:54.639514 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:54.642172 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":135,"threads_started":1}
W20260810 23:01:54.643935 30873 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:44491 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.56.62:44491: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:54.649631 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:54.644419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:57033)
0810 23:01:54.644521 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:54.644535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:54.644618 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:54.644840 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:54.644843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:54.644999 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:54.645093 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.645098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.645860 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.645864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.646414 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.646444 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.647067 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.647070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.647154 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.647688 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:54.647706 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:54.649490 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 23:01:54.650072 30879 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:54.644528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:57033 (local address 127.15.56.62:44491)
0810 23:01:54.644642 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:54.644647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:54.644660 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:01:54.644700 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:54.644703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:54.644745 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:54.644825 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:54.645205 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.645748 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.646571 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.646912 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.647040 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.649633 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:57033: BlockingRecv error: recv got EOF from 127.15.56.1:57033 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:54.655491 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:54.650221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:35943)
0810 23:01:54.650329 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:54.650342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:54.650415 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:54.650804 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:54.650807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:54.650969 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:54.651059 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.651065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.651811 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.651814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.652663 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.652672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.653107 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.653112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.653195 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.653625 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:54.653639 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:54.655364 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:01:54.655735 30873 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:01:54.655593 30879 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:54.650487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:35943 (local address 127.15.56.62:44491)
0810 23:01:54.650597 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:54.650601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:54.650614 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:01:54.650651 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:54.650654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:54.650696 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:54.650756 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:54.651183 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.651703 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.652795 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.653004 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.653075 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.655491 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:35943: BlockingRecv error: recv got EOF from 127.15.56.1:35943 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 23:01:54.759838 30880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:54.760160 30880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:54.760205 30880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:54.764396 30880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:54.764468 30880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:54.764534 30880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:01:54.770128 30880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.2.keytab
--principal=oryx/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:41081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=38387
--tserver_master_addrs=127.15.56.62:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:54.771409 30880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:54.772672 30880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:54.782711 30892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:54.782713 30889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:54.782714 30890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:54.783006 30880 server_base.cc:1034] running on GCE node
I20260810 23:01:54.783524 30880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:54.784214 30880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:54.785377 30880 hybrid_clock.cc:648] HybridClock initialized: now 1786402914785352 us; error 35 us; skew 500 ppm
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402914, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:54.788242 30880 init.cc:377] Logged in from keytab as oryx/127.15.56.2@KRBTEST.COM (short username oryx)
I20260810 23:01:54.789211 30880 webserver.cc:460] Webserver started at http://127.15.56.2:38387/ using document root <none> and password file <none>
I20260810 23:01:54.789806 30880 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:54.789896 30880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:54.794054 30880 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:01:54.796378 30904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:54.797724 30880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:54.797870 30880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "cf74055436c943678907f51e395fa115"
format_stamp: "Formatted at 2026-08-10 23:01:53 on dist-test-slave-6bbx"
server_key: "f79bde24d4d844ac55e2fbe0ff336ea5"
server_key_iv: "c59c3ca47baf61da6f625ca7cecce5c8"
server_key_version: "encryptionkey@0"
I20260810 23:01:54.798468 30880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:54.815369 30880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:54.818770 30880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:54.818969 30880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:54.819629 30880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:54.820623 30880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:54.820673 30880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:54.820720 30880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:54.820750 30880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:54.829756 30880 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:41081
I20260810 23:01:54.829792 31090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:41081 every 8 connection(s)
I20260810 23:01:54.830824 30880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:01:54.834074 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 30880
I20260810 23:01:54.834498 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:35765
--local_ip_for_outbound_sockets=127.15.56.3
--tserver_master_addrs=127.15.56.62:44491
--webserver_port=35237
--webserver_interface=127.15.56.3
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.56.3
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:01:54.843184 30879 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:54.832720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:39433 (local address 127.15.56.62:44491)
0810 23:01:54.832873 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:54.832877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:54.833758 (+ 881us) server_negotiation.cc:407] Connection header received
0810 23:01:54.835163 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:54.835168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:54.835234 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:54.835338 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:54.836856 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.837429 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.838655 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.838996 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.839131 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.843027 (+ 3896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:39433: BlockingRecv error: recv got EOF from 127.15.56.2:39433 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:01:54.843793 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:54.833124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:39433)
0810 23:01:54.833603 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:54.833649 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:54.834931 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:54.835501 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:54.835509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:54.835965 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:54.836673 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.836693 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.837563 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.837567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.838512 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.838521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.839164 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.839167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.839334 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.840009 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:54.840028 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:54.842826 (+ 2798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":136,"threads_started":1}
W20260810 23:01:54.844458 31091 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:44491 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.56.62:44491: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:54.850535 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:54.845050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:34281)
0810 23:01:54.845180 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:54.845195 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:54.845287 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:54.845526 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:54.845529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:54.845714 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:54.845821 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.845827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.846584 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.846587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.847166 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.847175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.847647 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.847651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.847773 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.848353 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:54.848367 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:54.850389 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:01:54.850675 30879 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:54.845165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:34281 (local address 127.15.56.62:44491)
0810 23:01:54.845321 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:54.845325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:54.845342 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:01:54.845388 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:54.845391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:54.845428 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:54.845493 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:54.845951 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.846472 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.847301 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.847535 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.847628 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.850548 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:34281: BlockingRecv error: recv got EOF from 127.15.56.2:34281 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:54.856564 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:54.851254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:44635)
0810 23:01:54.851366 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:54.851381 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:54.851453 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:54.851695 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:54.851698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:54.851834 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:54.851914 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.851919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.852918 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.852921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.853456 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:54.853465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.854029 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.854032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:54.854102 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.854587 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:54.854597 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:54.856451 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:01:54.856789 31091 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:01:54.856987 30879 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:54.851327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:44635 (local address 127.15.56.62:44491)
0810 23:01:54.851452 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:54.851456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:54.851470 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:01:54.851514 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:54.851517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:54.851562 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:54.851644 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:54.852013 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.852789 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.853555 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:54.853905 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:54.854003 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:54.856617 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:44635: BlockingRecv error: recv got EOF from 127.15.56.2:44635 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 23:01:54.956316 31094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:54.956584 31094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:54.956626 31094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:54.960566 31094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:54.960618 31094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:54.960675 31094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:01:54.965955 31094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/krb5kdc/oryx_127.15.56.3.keytab
--principal=oryx/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:35765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=35237
--tserver_master_addrs=127.15.56.62:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:54.966970 31094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:54.968009 31094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:54.978313 31099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:54.978349 31102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:54.978322 31100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:54.979185 31094 server_base.cc:1034] running on GCE node
I20260810 23:01:54.979612 31094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:54.980523 31094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:54.981720 31094 hybrid_clock.cc:648] HybridClock initialized: now 1786402914981700 us; error 39 us; skew 500 ppm
Aug 10 23:01:54 dist-test-slave-6bbx krb5kdc[29673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402914, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:54.984815 31094 init.cc:377] Logged in from keytab as oryx/127.15.56.3@KRBTEST.COM (short username oryx)
I20260810 23:01:54.985785 31094 webserver.cc:460] Webserver started at http://127.15.56.3:35237/ using document root <none> and password file <none>
I20260810 23:01:54.986410 31094 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:54.986505 31094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:54.990913 31094 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:01:54.993346 31115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:54.994678 31094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:01:54.994798 31094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "dd48f69f13544947a146705758fe4bef"
format_stamp: "Formatted at 2026-08-10 23:01:54 on dist-test-slave-6bbx"
server_key: "4dc567ae6274d0d0801de0963930fcc1"
server_key_iv: "db7f99631026ca140e212c265678890b"
server_key_version: "encryptionkey@0"
I20260810 23:01:54.995370 31094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:55.008343 31094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:55.011636 31094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:55.011886 31094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:55.012589 31094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:01:55.013492 31094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:01:55.013541 31094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:55.013581 31094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:01:55.013597 31094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:55.022205 31094 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:35765
I20260810 23:01:55.022233 31298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:35765 every 8 connection(s)
I20260810 23:01:55.023231 31094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
Aug 10 23:01:55 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:55.033018 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 31094
Aug 10 23:01:55 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:01:55.034340 30879 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:55.025063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:53889 (local address 127.15.56.62:44491)
0810 23:01:55.025195 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:55.025199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:55.026056 (+ 857us) server_negotiation.cc:407] Connection header received
0810 23:01:55.027415 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:55.027418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:55.027475 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:55.027553 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:55.028784 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.029386 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.030531 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.030749 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.030846 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:55.034218 (+ 3372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:53889: BlockingRecv error: recv got EOF from 127.15.56.3:53889 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:01:55.034989 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:55.025450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:53889)
0810 23:01:55.025906 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:55.025951 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:55.027187 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:55.027777 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:55.027784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:55.028143 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:55.028637 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:55.028651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.029518 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.029521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:55.030408 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:55.030418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.030893 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.030897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:55.031061 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:55.031623 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:55.031641 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:55.034107 (+ 2466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
W20260810 23:01:55.035506 31301 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:44491 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.56.62:44491: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Aug 10 23:01:55 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:55 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:55.040807 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:55.035941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:36977)
0810 23:01:55.036029 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:55.036041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:55.036113 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:55.036303 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:55.036306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:55.036458 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:55.036554 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:55.036560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.037253 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.037257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:55.037792 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:55.037800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.038220 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.038224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:55.038320 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:55.038717 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:55.038730 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:55.040671 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 23:01:55.040979 30879 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:55.036025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:36977 (local address 127.15.56.62:44491)
0810 23:01:55.036127 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:55.036131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:55.036146 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:01:55.036180 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:55.036183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:55.036216 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:55.036278 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:55.036659 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.037157 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.037907 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.038124 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.038207 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:55.040848 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:36977: BlockingRecv error: recv got EOF from 127.15.56.3:36977 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 23:01:55.043941 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:55.034077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33538 (local address 127.15.56.62:44491)
0810 23:01:55.034312 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:55.034315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:55.034326 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:01:55.034355 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:55.034357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:55.034392 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:55.035223 (+ 831us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:55.035348 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.036010 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.036933 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.037167 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.037255 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:55.038665 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:55.038673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:55.038696 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:55.038729 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:55.041420 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:55.042011 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:55.042019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:55.042028 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:55.042185 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:55.042557 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:55.042561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:55.042563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:55.042976 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:55.043192 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:55.043500 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 23:01:55.043807 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
Aug 10 23:01:55 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:55 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:55.047745 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:55.041643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:56329)
0810 23:01:55.041808 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:55.041832 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:55.041918 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:55.042233 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:55.042237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:55.042380 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:55.042468 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:55.042473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.043537 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.043542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:55.044410 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:55.044423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.044980 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.044984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:55.045072 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:55.045541 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:55.045555 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:55.047579 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
W20260810 23:01:55.047887 30879 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:55.041708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:56329 (local address 127.15.56.62:44491)
0810 23:01:55.041842 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:55.041846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:55.041875 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:01:55.042047 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:55.042050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:55.042118 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:55.042200 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:55.042626 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.043394 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.044530 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.044866 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.044967 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:55.047757 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:56329: BlockingRecv error: recv got EOF from 127.15.56.3:56329 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 23:01:55.047984 31301 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 23:01:55 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:55 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:55.661578 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:55.656379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:58807)
0810 23:01:55.656645 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:55.656658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:55.656751 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:55.656975 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:55.656977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:55.657159 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:55.657259 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:55.657264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.657958 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.657960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:55.658545 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:55.658552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.658922 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.658924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:55.658983 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:55.659500 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:55.659512 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:55.661442 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
W20260810 23:01:55.661759 31313 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:55.656476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:58807 (local address 127.15.56.62:44491)
0810 23:01:55.656731 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:55.656735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:55.656748 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:01:55.656829 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:55.656831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:55.656874 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:55.656951 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:55.657344 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.657875 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.658630 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.658844 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.658914 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:55.661610 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:58807: BlockingRecv error: recv got EOF from 127.15.56.1:58807 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
Aug 10 23:01:55 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:55 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:55.863261 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:55.857503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:55877)
0810 23:01:55.857798 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:55.857813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:55.857922 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:55.858134 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:55.858136 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:55.858338 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:55.858442 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:55.858448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.859183 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.859186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:55.859876 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:55.859889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.860436 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.860440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:55.860520 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:55.861094 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:55.861107 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:55.863118 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
W20260810 23:01:55.863389 31313 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:55.857642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:55877 (local address 127.15.56.62:44491)
0810 23:01:55.857801 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:55.857805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:55.857880 (+ 75us) server_negotiation.cc:407] Connection header received
0810 23:01:55.857996 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:55.857998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:55.858042 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:55.858109 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:55.858530 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.859066 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.860038 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:55.860282 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:55.860386 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:55.863270 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:55877: BlockingRecv error: recv got EOF from 127.15.56.2:55877 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 23:01:56 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:56 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:56.054001 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:56.048649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:40637)
0810 23:01:56.048939 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:56.048952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:56.049057 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:56.049273 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:56.049275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:56.049463 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:56.049588 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:56.049595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.050284 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.050286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:56.050870 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:56.050878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.051311 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.051314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:56.051403 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:56.051939 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:56.051952 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:56.053859 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
W20260810 23:01:56.054207 31313 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:56.048757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:40637 (local address 127.15.56.62:44491)
0810 23:01:56.048906 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:56.048910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:56.049018 (+ 108us) server_negotiation.cc:407] Connection header received
0810 23:01:56.049127 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:56.049129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:56.049175 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:56.049246 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:56.049680 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.050203 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.050957 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.051177 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.051251 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:56.054049 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:40637: BlockingRecv error: recv got EOF from 127.15.56.3:40637 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 23:01:56 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:56 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:56.667706 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:56.662349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:35655)
0810 23:01:56.662676 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:56.662691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:56.662798 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:56.663005 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:56.663007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:56.663201 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:56.663303 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:56.663309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.664107 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.664110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:56.664689 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:56.664699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.665135 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.665137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:56.665196 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:56.665678 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:56.665689 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:56.667514 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
W20260810 23:01:56.667800 31317 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:56.662424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:35655 (local address 127.15.56.62:44491)
0810 23:01:56.662675 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:56.662678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:56.662745 (+ 67us) server_negotiation.cc:407] Connection header received
0810 23:01:56.662863 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:56.662865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:56.662905 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:56.662977 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:56.663396 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.663979 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.664808 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.665046 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.665125 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:56.667655 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:35655: BlockingRecv error: recv got EOF from 127.15.56.1:35655 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
Aug 10 23:01:56 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:56 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:56.873183 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:56.864081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:51703)
0810 23:01:56.864367 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:56.864381 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:56.864484 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:56.864678 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:56.864680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:56.864881 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:56.864985 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:56.864990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.865721 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.865723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:56.866278 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:56.866284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.866632 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.866635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:56.866695 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:56.870915 (+ 4220us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:56.870928 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:56.872998 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
W20260810 23:01:56.873257 31317 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:56.864171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:51703 (local address 127.15.56.62:44491)
0810 23:01:56.864312 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:56.864325 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:56.864434 (+ 109us) server_negotiation.cc:407] Connection header received
0810 23:01:56.864547 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:56.864549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:56.864592 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:56.864653 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:56.865121 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.865642 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.866359 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:56.866559 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:56.866626 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:56.873150 (+ 6524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:51703: BlockingRecv error: recv got EOF from 127.15.56.2:51703 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 23:01:57 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:57 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:57.060389 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:57.054746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:43767)
0810 23:01:57.055092 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:57.055106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:57.055216 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:57.055492 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:57.055495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:57.055756 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:57.055867 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:57.055874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.056683 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.056686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:57.057343 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:57.057353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.057742 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.057745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:57.057806 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:57.058264 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:57.058276 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:57.060269 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
W20260810 23:01:57.060495 31317 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:57.054943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:43767 (local address 127.15.56.62:44491)
0810 23:01:57.055094 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:57.055097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:57.055183 (+ 86us) server_negotiation.cc:407] Connection header received
0810 23:01:57.055295 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:57.055297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:57.055344 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:57.055410 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:57.055994 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.056535 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.057446 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.057654 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.057724 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:57.060395 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:43767: BlockingRecv error: recv got EOF from 127.15.56.3:43767 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 23:01:57 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:57 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:57.674340 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:57.668601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:60779)
0810 23:01:57.668921 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:57.668934 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:57.669029 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:57.669244 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:57.669246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:57.669444 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:57.669569 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:57.669577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.670621 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.670626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:57.671352 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:57.671363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.671866 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.671869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:57.671936 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:57.672431 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:57.672444 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:57.674193 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":123,"threads_started":1}
W20260810 23:01:57.674535 31324 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:57.668721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:60779 (local address 127.15.56.62:44491)
0810 23:01:57.668986 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:57.668990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:57.669004 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:01:57.669098 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:57.669100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:57.669143 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:57.669228 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:57.669734 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.670475 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.671464 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.671748 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.671844 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:57.674357 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:60779: BlockingRecv error: recv got EOF from 127.15.56.1:60779 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
Aug 10 23:01:57 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:57 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:57.879581 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:57.873923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:57857)
0810 23:01:57.874225 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:57.874240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:57.874383 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:57.874630 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:57.874633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:57.874848 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:57.874951 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:57.874958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.875943 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.875947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:57.876519 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:57.876527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.876972 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.876976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:57.877058 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:57.877558 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:57.877569 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:57.879430 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
W20260810 23:01:57.879712 31324 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:57.873998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:57857 (local address 127.15.56.62:44491)
0810 23:01:57.874181 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:57.874186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:57.874320 (+ 134us) server_negotiation.cc:407] Connection header received
0810 23:01:57.874473 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:57.874477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:57.874525 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:57.874593 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:57.875069 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.875830 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.876634 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:57.876868 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:57.876956 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:57.879574 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:57857: BlockingRecv error: recv got EOF from 127.15.56.2:57857 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 23:01:58 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:58 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:58.067127 31326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:58.061421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:47941)
0810 23:01:58.061717 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:58.061734 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:58.061871 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:58.062160 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:58.062163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:58.062405 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:58.062515 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:58.062523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.063440 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.063444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:58.064119 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:58.064130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.064538 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.064540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:58.064598 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:58.065085 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:58.065096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:58.067001 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
W20260810 23:01:58.067344 31324 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:58.061581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:47941 (local address 127.15.56.62:44491)
0810 23:01:58.061732 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:58.061737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:58.061841 (+ 104us) server_negotiation.cc:407] Connection header received
0810 23:01:58.061956 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:58.061960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:58.062017 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:58.062119 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:58.062651 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.063325 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.064226 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.064450 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.064529 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:58.067219 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:47941: BlockingRecv error: recv got EOF from 127.15.56.3:47941 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 23:01:58 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:58 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:58.680641 31327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:58.675125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:47469)
0810 23:01:58.675387 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:58.675400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:58.675495 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:58.675819 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:58.675822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:58.676005 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:58.676110 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:58.676115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.676921 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.676924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:58.677510 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:58.677519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.677923 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.677926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:58.677990 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:58.678483 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:58.678495 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:58.680515 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
W20260810 23:01:58.680759 31329 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:58.675229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:47469 (local address 127.15.56.62:44491)
0810 23:01:58.675474 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:58.675478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:58.675491 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:01:58.675574 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:58.675576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:58.675617 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:58.675692 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:58.676205 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.676760 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.677610 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.677833 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.677907 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:58.680631 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:47469: BlockingRecv error: recv got EOF from 127.15.56.1:47469 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
Aug 10 23:01:58 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:58 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:58.886294 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:58.880516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:51789)
0810 23:01:58.880822 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:58.880839 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:58.880973 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:58.881225 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:58.881228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:58.881455 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:58.881561 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:58.881568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.882358 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.882361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:58.882909 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:58.882916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.883357 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.883361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:58.883445 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:58.884069 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:58.884082 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:58.886163 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":92,"threads_started":1}
W20260810 23:01:58.886423 31329 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:58.880630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:51789 (local address 127.15.56.62:44491)
0810 23:01:58.880854 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:58.880860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:58.880929 (+ 69us) server_negotiation.cc:407] Connection header received
0810 23:01:58.881067 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:58.881070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:58.881117 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:58.881187 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:58.881693 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.882245 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.883029 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:58.883249 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:58.883337 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:58.886297 (+ 2960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:51789: BlockingRecv error: recv got EOF from 127.15.56.2:51789 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 10 23:01:59 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:59 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:59.074151 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:59.068087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:41423)
0810 23:01:59.068373 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:59.068388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:59.068495 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:59.068742 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:59.068744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:59.068947 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:59.069046 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:59.069051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.069822 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.069825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:59.070451 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:59.070461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.071017 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.071022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:59.071113 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:59.071699 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:59.071718 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:59.073955 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
W20260810 23:01:59.074247 31329 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:59.068210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:41423 (local address 127.15.56.62:44491)
0810 23:01:59.068344 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:59.068347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:59.068452 (+ 105us) server_negotiation.cc:407] Connection header received
0810 23:01:59.068599 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:59.068602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:59.068649 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:59.068718 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:59.069144 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.069691 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.070581 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.070840 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.070922 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:59.074122 (+ 3200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:41423: BlockingRecv error: recv got EOF from 127.15.56.3:41423 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 23:01:59 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:59 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:59.686980 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:59.681471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:38789)
0810 23:01:59.681748 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:59.681761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:59.681849 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:59.682078 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:59.682080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:59.682284 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:59.682400 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:59.682406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.683208 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.683211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:59.683870 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:59.683880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.684309 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.684312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:59.684394 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:59.684904 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:59.684916 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:59.686819 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
W20260810 23:01:59.687098 31335 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:59.681574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:38789 (local address 127.15.56.62:44491)
0810 23:01:59.681837 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:59.681841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:59.681855 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:01:59.681930 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:59.681932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:59.681973 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:59.682056 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:59.682523 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.683096 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.683978 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.684206 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.684288 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:59.686974 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:38789: BlockingRecv error: recv got EOF from 127.15.56.1:38789 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
Aug 10 23:01:59 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:01:59 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:01:59.893678 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:59.887402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:51497)
0810 23:01:59.887722 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:59.887740 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:59.887845 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:59.888154 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:59.888158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:59.888368 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:59.888476 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:59.888482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.889431 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.889435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:59.890131 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:59.890143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.890740 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.890745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:59.890852 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:59.891429 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:59.891442 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:59.893531 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
W20260810 23:01:59.893837 31335 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:59.887645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:51497 (local address 127.15.56.62:44491)
0810 23:01:59.887853 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:59.887858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:59.887877 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:01:59.887926 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:59.887928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:59.887971 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:59.888062 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:59.888623 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.889268 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.890294 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:59.890591 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:59.890690 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:59.893690 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:51497: BlockingRecv error: recv got EOF from 127.15.56.2:51497 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 10 23:02:00 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:00 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:00.082995 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:00.076686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:51751)
0810 23:02:00.077060 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:00.077077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:00.077211 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:00.077551 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:00.077554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:00.077759 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:00.077871 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:00.077877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.078745 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.078749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:00.079311 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:00.079329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.079962 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.079965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:00.080037 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:00.080686 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:00.080698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:00.082861 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":105,"threads_started":1}
W20260810 23:02:00.083144 31335 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:00.076829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:51751 (local address 127.15.56.62:44491)
0810 23:02:00.076996 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:00.077001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:00.077183 (+ 182us) server_negotiation.cc:407] Connection header received
0810 23:02:00.077301 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:00.077304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:00.077359 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:00.077447 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:00.078044 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.078628 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.079457 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.079825 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.079943 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:00.083025 (+ 3082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:51751: BlockingRecv error: recv got EOF from 127.15.56.3:51751 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 23:02:00 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:00 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:00.693722 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:00.688004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:54395)
0810 23:02:00.688296 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:00.688309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:00.688407 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:00.688628 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:00.688630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:00.688838 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:00.688943 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:00.688949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.689807 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.689810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:00.690381 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:00.690389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.690907 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.690910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:00.690989 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:00.691545 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:00.691559 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:00.693591 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
W20260810 23:02:00.693882 31340 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:00.688085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:54395 (local address 127.15.56.62:44491)
0810 23:02:00.688360 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:00.688364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:00.688381 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:02:00.688486 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:00.688488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:00.688533 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:00.688614 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:00.689087 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.689680 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.690520 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.690763 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.690857 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:00.693753 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:54395: BlockingRecv error: recv got EOF from 127.15.56.1:54395 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":97,"threads_started":1}
Aug 10 23:02:00 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:00 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:00.900276 31343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:00.894576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:54295)
0810 23:02:00.894868 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:00.894883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:00.894992 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:00.895221 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:00.895223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:00.895449 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:00.895554 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:00.895560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.896337 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.896340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:00.896910 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:00.896920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.897371 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.897373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:00.897434 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:00.898005 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:00.898018 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:00.900112 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
W20260810 23:02:00.900364 31340 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:00.894699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:54295 (local address 127.15.56.62:44491)
0810 23:02:00.894865 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:00.894869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:00.894950 (+ 81us) server_negotiation.cc:407] Connection header received
0810 23:02:00.895065 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:00.895068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:00.895112 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:00.895186 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:00.895677 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.896210 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.897033 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:00.897264 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:00.897357 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:00.900241 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:54295: BlockingRecv error: recv got EOF from 127.15.56.2:54295 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 23:02:01 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:01 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:01.089465 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:01.083833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:33747)
0810 23:02:01.084109 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:01.084124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:01.084227 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:01.084435 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:01.084437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:01.084619 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:01.084718 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:01.084723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.085489 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.085492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:01.086063 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:01.086072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.086537 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.086539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:01.086600 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:01.087082 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:01.087094 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:01.089309 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":106,"threads_started":1}
W20260810 23:02:01.089581 31340 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:01.083889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:33747 (local address 127.15.56.62:44491)
0810 23:02:01.084023 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:01.084027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:01.084222 (+ 195us) server_negotiation.cc:407] Connection header received
0810 23:02:01.084297 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:01.084299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:01.084350 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:01.084423 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:01.084830 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.085392 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.086191 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.086438 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.086529 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:01.089462 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:33747: BlockingRecv error: recv got EOF from 127.15.56.3:33747 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 23:02:01 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:01 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:01.699987 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:01.694577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:40501)
0810 23:02:01.694906 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:01.694922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:01.695047 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:01.695349 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:01.695353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:01.695555 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:01.695657 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:01.695718 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.696436 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.696438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:01.697002 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:01.697009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.697427 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.697429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:01.697486 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:01.697986 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:01.697998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:01.699831 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":75,"threads_started":1}
W20260810 23:02:01.700133 31347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:01.694629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:40501 (local address 127.15.56.62:44491)
0810 23:02:01.694893 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:01.694898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:01.694998 (+ 100us) server_negotiation.cc:407] Connection header received
0810 23:02:01.695148 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:01.695151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:01.695209 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:01.695302 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:01.695800 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.696354 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.697098 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.697344 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.697420 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:01.699993 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:40501: BlockingRecv error: recv got EOF from 127.15.56.1:40501 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
Aug 10 23:02:01 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:01 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:01.909637 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:01.901116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:57669)
0810 23:02:01.901427 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:01.901441 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:01.901548 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:01.901774 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:01.901776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:01.901999 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:01.902125 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:01.902133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.903003 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.903011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:01.903616 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:01.903626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.904105 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.904108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:01.904204 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:01.907548 (+ 3344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:01.907561 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:01.909511 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":89,"threads_started":1}
W20260810 23:02:01.909817 31347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:01.901207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:57669 (local address 127.15.56.62:44491)
0810 23:02:01.901375 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:01.901380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:01.901549 (+ 169us) server_negotiation.cc:407] Connection header received
0810 23:02:01.901623 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:01.901625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:01.901675 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:01.901748 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:01.902271 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.902869 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.903755 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:01.903983 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:01.904066 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:01.909686 (+ 5620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:57669: BlockingRecv error: recv got EOF from 127.15.56.2:57669 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 23:02:02 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:02 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:02.095997 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:02.090377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:37215)
0810 23:02:02.090737 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:02.090750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:02.090873 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:02.091125 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:02.091127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:02.091346 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:02.091445 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:02.091452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.092262 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.092266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:02.092813 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:02.092829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.093259 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.093262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:02.093344 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:02.093847 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:02.093860 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:02.095838 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":94,"threads_started":1}
W20260810 23:02:02.096062 31347 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:02.090493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:37215 (local address 127.15.56.62:44491)
0810 23:02:02.090632 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:02.090637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:02.090825 (+ 188us) server_negotiation.cc:407] Connection header received
0810 23:02:02.090965 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:02.090968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:02.091017 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:02.091083 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:02.091563 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.092155 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.092933 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.093155 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.093242 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:02.095960 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:37215: BlockingRecv error: recv got EOF from 127.15.56.3:37215 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 23:02:02 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:02 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:02.706545 31354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:02.700775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:36917)
0810 23:02:02.701042 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:02.701056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:02.701155 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:02.701467 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:02.701470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:02.701668 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:02.701812 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:02.701820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.702635 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.702638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:02.703347 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:02.703359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.703923 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.703927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:02.704012 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:02.704572 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:02.704588 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:02.706421 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
W20260810 23:02:02.706715 31355 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:02.700896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:36917 (local address 127.15.56.62:44491)
0810 23:02:02.701121 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:02.701125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:02.701139 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:02:02.701238 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:02.701241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:02.701301 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:02.701395 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:02.701933 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.702522 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.703482 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.703779 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.703911 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:02.706578 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:36917: BlockingRecv error: recv got EOF from 127.15.56.1:36917 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
Aug 10 23:02:02 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:02 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:02.916092 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:02.910507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:36909)
0810 23:02:02.910879 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:02.910892 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:02.911000 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:02.911216 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:02.911218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:02.911419 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:02.911522 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:02.911527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.912472 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.912475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:02.913046 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:02.913055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.913489 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.913491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:02.913550 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:02.914067 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:02.914080 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:02.915966 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":84,"threads_started":1}
W20260810 23:02:02.916291 31355 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:02.910606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:36909 (local address 127.15.56.62:44491)
0810 23:02:02.910772 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:02.910777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:02.910962 (+ 185us) server_negotiation.cc:407] Connection header received
0810 23:02:02.911082 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:02.911084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:02.911131 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:02.911217 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:02.911612 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.912361 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.913154 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:02.913401 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:02.913479 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:02.916160 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:36909: BlockingRecv error: recv got EOF from 127.15.56.2:36909 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 23:02:03 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:03 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:03.103837 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:03.097231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:42247)
0810 23:02:03.097570 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:03.097585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:03.097707 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:03.098068 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:03.098071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:03.098274 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:03.098387 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:03.098393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.099432 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.099436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:03.100256 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:03.100270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.100892 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.100896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:03.100980 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:03.101553 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:03.101568 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:03.103636 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":91,"threads_started":1}
W20260810 23:02:03.103962 31355 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:03.097415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:42247 (local address 127.15.56.62:44491)
0810 23:02:03.097573 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:03.097577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:03.097661 (+ 84us) server_negotiation.cc:407] Connection header received
0810 23:02:03.097846 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:03.097850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:03.097906 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:03.097995 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:03.098540 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.099287 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.100446 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.100779 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.100880 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:03.103850 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:42247: BlockingRecv error: recv got EOF from 127.15.56.3:42247 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 23:02:03 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:03 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:03.713274 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:03.707417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:57431)
0810 23:02:03.707744 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:03.707761 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:03.707875 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:03.708176 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:03.708178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:03.708370 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:03.708479 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:03.708485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.709341 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.709344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:03.709947 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:03.709957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.710522 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.710526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:03.710615 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:03.711241 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:03.711254 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:03.713148 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
W20260810 23:02:03.713482 31365 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:03.707552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:57431 (local address 127.15.56.62:44491)
0810 23:02:03.707868 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:03.707872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:03.707888 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:02:03.707987 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:03.707990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:03.708035 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:03.708116 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:03.708619 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.709196 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.710078 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.710396 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.710484 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:03.713329 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:57431: BlockingRecv error: recv got EOF from 127.15.56.1:57431 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":125,"threads_started":1}
Aug 10 23:02:03 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:03 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:03.922359 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:03.916906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:42817)
0810 23:02:03.917194 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:03.917208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:03.917315 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:03.917530 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:03.917532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:03.917727 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:03.917831 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:03.917836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.918613 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.918616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:03.919184 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:03.919192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.919589 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.919592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:03.919715 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:03.920227 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:03.920240 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:03.922235 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
W20260810 23:02:03.922511 31365 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:03.917038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:42817 (local address 127.15.56.62:44491)
0810 23:02:03.917206 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:03.917209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:03.917293 (+ 84us) server_negotiation.cc:407] Connection header received
0810 23:02:03.917387 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:03.917389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:03.917435 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:03.917501 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:03.917942 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.918499 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.919286 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:03.919497 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:03.919567 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:03.922391 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:42817: BlockingRecv error: recv got EOF from 127.15.56.2:42817 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 23:02:04 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:04 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:04.110203 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:04.104644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:38981)
0810 23:02:04.104982 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:04.104998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:04.105119 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:04.105367 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:04.105370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:04.105590 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:04.105692 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:04.105699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.106475 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.106479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:04.107076 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:04.107088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.107537 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.107541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:04.107630 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:04.108175 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:04.108191 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:04.110046 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":103,"threads_started":1}
W20260810 23:02:04.110311 31365 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:04.104772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:38981 (local address 127.15.56.62:44491)
0810 23:02:04.104954 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:04.104959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:04.105070 (+ 111us) server_negotiation.cc:407] Connection header received
0810 23:02:04.105208 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:04.105211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:04.105261 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:04.105327 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:04.105811 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.106367 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.107189 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.107428 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.107518 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:04.110202 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:38981: BlockingRecv error: recv got EOF from 127.15.56.3:38981 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 23:02:04 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:04 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:04.719276 31369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:04.714025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:55781)
0810 23:02:04.714297 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:04.714310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:04.714397 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:04.714619 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:04.714622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:04.714817 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:04.714917 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:04.714922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.715648 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.715652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:04.716303 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:04.716314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.716730 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.716734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:04.716811 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:04.717300 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:04.717312 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:04.719144 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
W20260810 23:02:04.719444 31370 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:04.714113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:55781 (local address 127.15.56.62:44491)
0810 23:02:04.714351 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:04.714355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:04.714368 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:02:04.714474 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:04.714477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:04.714518 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:04.714591 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:04.715002 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.715530 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.716397 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.716614 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.716684 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:04.719287 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:55781: BlockingRecv error: recv got EOF from 127.15.56.1:55781 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
Aug 10 23:02:04 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:04 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:04.928911 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:04.923382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:41285)
0810 23:02:04.923728 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:04.923746 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:04.923871 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:04.924117 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:04.924120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:04.924336 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:04.924435 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:04.924441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.925192 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.925196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:04.925753 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:04.925761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.926202 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.926205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:04.926286 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:04.926807 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:04.926831 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:04.928776 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
W20260810 23:02:04.929127 31370 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:04.923512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:41285 (local address 127.15.56.62:44491)
0810 23:02:04.923729 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:04.923734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:04.923819 (+ 85us) server_negotiation.cc:407] Connection header received
0810 23:02:04.923963 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:04.923967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:04.924013 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:04.924078 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:04.924555 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.925084 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.925872 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:04.926098 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:04.926188 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:04.928964 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:41285: BlockingRecv error: recv got EOF from 127.15.56.2:41285 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 10 23:02:05 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:05 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:05.120692 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:05.111097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:57743)
0810 23:02:05.111414 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:05.111429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:05.111538 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:05.111844 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:05.111847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:05.112038 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:05.112141 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:05.112148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.112903 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.112906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:05.114842 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:05.114865 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.116037 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.116041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:05.116126 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:05.116788 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:05.116804 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:05.120517 (+ 3713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":85,"threads_started":1}
W20260810 23:02:05.120810 31370 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:05.111281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:57743 (local address 127.15.56.62:44491)
0810 23:02:05.111421 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:05.111425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:05.111496 (+ 71us) server_negotiation.cc:407] Connection header received
0810 23:02:05.111611 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:05.111613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:05.111656 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:05.111751 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:05.112248 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.112774 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.115367 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.115643 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.115900 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:05.120692 (+ 4792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:57743: BlockingRecv error: recv got EOF from 127.15.56.3:57743 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 23:02:05 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:05 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:05.726334 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:05.720363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:47851)
0810 23:02:05.720645 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:05.720661 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:05.720764 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:05.721065 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:05.721069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:05.721344 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:05.721466 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:05.721473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.722334 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.722337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:05.722925 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:05.722934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.723469 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.723473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:05.723561 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:05.724178 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:05.724192 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:05.726151 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
W20260810 23:02:05.726451 31378 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:05.720534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:47851 (local address 127.15.56.62:44491)
0810 23:02:05.720777 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:05.720782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:05.720798 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:02:05.720842 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:05.720845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:05.720898 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:05.720996 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:05.721591 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.722208 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.723070 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.723324 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.723417 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:05.726319 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:47851: BlockingRecv error: recv got EOF from 127.15.56.1:47851 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
Aug 10 23:02:05 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:05 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:05.935497 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:05.929808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:43739)
0810 23:02:05.930105 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:05.930118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:05.930230 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:05.930466 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:05.930468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:05.930663 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:05.930766 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:05.930771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.931527 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.931530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:05.932169 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:05.932181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.932624 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.932627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:05.932697 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:05.933263 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:05.933275 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:05.935297 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
W20260810 23:02:05.935599 31378 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:05.929910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:43739 (local address 127.15.56.62:44491)
0810 23:02:05.930044 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:05.930048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:05.930184 (+ 136us) server_negotiation.cc:407] Connection header received
0810 23:02:05.930308 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:05.930311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:05.930359 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:05.930434 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:05.930862 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.931403 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.932288 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:05.932521 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:05.932600 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:05.935483 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:43739: BlockingRecv error: recv got EOF from 127.15.56.2:43739 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 23:02:06 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:06 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:06.127072 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:06.121576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:33497)
0810 23:02:06.121963 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:06.121976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:06.122076 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:06.122308 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:06.122309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:06.122503 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:06.122603 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:06.122608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.123409 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.123412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:06.124016 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:06.124025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.124446 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.124448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:06.124507 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:06.125067 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:06.125080 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:06.126947 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":214,"threads_started":1}
W20260810 23:02:06.127202 31378 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:06.121650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:33497 (local address 127.15.56.62:44491)
0810 23:02:06.121836 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:06.121841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:06.122042 (+ 201us) server_negotiation.cc:407] Connection header received
0810 23:02:06.122159 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:06.122162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:06.122215 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:06.122286 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:06.122746 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.123295 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.124118 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.124352 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.124437 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:06.127089 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:33497: BlockingRecv error: recv got EOF from 127.15.56.3:33497 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 23:02:06 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:06 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:06.733476 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:06.727219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:45499)
0810 23:02:06.727519 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:06.727533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:06.727647 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:06.728085 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:06.728088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:06.728289 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:06.728406 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:06.728412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.729257 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.729261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:06.729849 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:06.729859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.730349 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.730352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:06.730429 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:06.731041 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:06.731058 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:06.733311 (+ 2253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":90,"threads_started":1}
W20260810 23:02:06.733629 31387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:06.727308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:45499 (local address 127.15.56.62:44491)
0810 23:02:06.727548 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:06.727552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:06.727588 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:02:06.727819 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:06.727822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:06.727884 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:06.727989 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:06.728548 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.729109 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.729975 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.730219 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.730325 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:06.733504 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:45499: BlockingRecv error: recv got EOF from 127.15.56.1:45499 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
Aug 10 23:02:06 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:06 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:06.945590 31388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:06.936469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:59403)
0810 23:02:06.936752 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:06.936767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:06.936886 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:06.937158 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:06.937162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:06.937364 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:06.937466 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:06.937473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.938228 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.938232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:06.938785 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:06.938793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.939231 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.939234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:06.939317 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:06.943499 (+ 4182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:06.943512 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:06.945437 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
W20260810 23:02:06.945708 31387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:06.936586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:59403 (local address 127.15.56.62:44491)
0810 23:02:06.936751 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:06.936756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:06.936835 (+ 79us) server_negotiation.cc:407] Connection header received
0810 23:02:06.936972 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:06.936975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:06.937025 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:06.937093 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:06.937582 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.938121 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.938898 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:06.939129 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:06.939216 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:06.945595 (+ 6379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:59403: BlockingRecv error: recv got EOF from 127.15.56.2:59403 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 23:02:07 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:07 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:07.133749 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:07.127966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:56851)
0810 23:02:07.128264 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:07.128278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:07.128388 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:07.128676 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:07.128679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:07.128881 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:07.128987 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:07.128993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.129775 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.129778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:07.130371 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:07.130382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.130867 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.130870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:07.130950 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:07.131508 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:07.131521 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:07.133563 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
W20260810 23:02:07.133819 31387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:07.128096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:56851 (local address 127.15.56.62:44491)
0810 23:02:07.128256 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:07.128260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:07.128349 (+ 89us) server_negotiation.cc:407] Connection header received
0810 23:02:07.128463 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:07.128465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:07.128514 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:07.128591 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:07.129118 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.129652 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.130492 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.130732 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.130830 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:07.133713 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:56851: BlockingRecv error: recv got EOF from 127.15.56.3:56851 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 23:02:07 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:07 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:07.740062 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:07.734276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:32963)
0810 23:02:07.734581 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:07.734597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:07.734695 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:07.734990 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:07.734992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:07.735183 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:07.735284 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:07.735290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.736123 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.736126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:07.736699 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:07.736706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.737264 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.737267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:07.737356 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:07.737911 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:07.737924 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:07.739900 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":77,"threads_started":1}
W20260810 23:02:07.740203 31392 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:07.734383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:32963 (local address 127.15.56.62:44491)
0810 23:02:07.734643 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:07.734647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:07.734660 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:02:07.734822 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:07.734825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:07.734876 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:07.734962 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:07.735415 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.736032 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.736822 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.737066 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.737192 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:07.740050 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:32963: BlockingRecv error: recv got EOF from 127.15.56.1:32963 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
Aug 10 23:02:07 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:07 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:07.952366 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:07.946443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:38445)
0810 23:02:07.946765 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:07.946781 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:07.946905 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:07.947193 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:07.947196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:07.947404 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:07.947509 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:07.947516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.948345 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.948349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:07.948911 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:07.948919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.949356 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.949359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:07.949441 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:07.949960 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:07.949973 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:07.952211 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":98,"threads_started":1}
W20260810 23:02:07.952520 31392 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:07.946642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:38445 (local address 127.15.56.62:44491)
0810 23:02:07.946776 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:07.946781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:07.946854 (+ 73us) server_negotiation.cc:407] Connection header received
0810 23:02:07.946995 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:07.946999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:07.947046 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:07.947114 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:07.947650 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.948229 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.949022 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:07.949251 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:07.949341 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:07.952381 (+ 3040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:38445: BlockingRecv error: recv got EOF from 127.15.56.2:38445 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 23:02:08 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:08 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:08.140892 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:08.134846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:41251)
0810 23:02:08.135317 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:08.135334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:08.135471 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:08.135784 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:08.135787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:08.136064 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:08.136188 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:08.136196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.136991 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.136995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:08.137573 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:08.137583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.138055 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.138059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:08.138147 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:08.138711 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:08.138723 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:08.140758 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":142,"threads_started":1}
W20260810 23:02:08.141219 31392 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:08.135049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:41251 (local address 127.15.56.62:44491)
0810 23:02:08.135279 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:08.135284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:08.135417 (+ 133us) server_negotiation.cc:407] Connection header received
0810 23:02:08.135574 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:08.135578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:08.135635 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:08.135788 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:08.136318 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.136871 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.137695 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.137938 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.138031 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:08.140999 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:41251: BlockingRecv error: recv got EOF from 127.15.56.3:41251 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 10 23:02:08 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:08 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:08.751919 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:08.740892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:49849)
0810 23:02:08.741223 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:08.741239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:08.741339 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:08.741624 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:08.741628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:08.741838 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:08.741945 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:08.741951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.742729 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.742732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:08.743346 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:08.743357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.743914 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.743918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:08.743999 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:08.749489 (+ 5490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:08.749508 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:08.751725 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":97,"threads_started":1}
W20260810 23:02:08.752182 31397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:08.740981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:49849 (local address 127.15.56.62:44491)
0810 23:02:08.741274 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:08.741278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:08.741292 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:02:08.741422 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:08.741424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:08.741469 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:08.741543 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:08.742045 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.742600 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.743483 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.743778 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.743857 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:08.751931 (+ 8074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:49849: BlockingRecv error: recv got EOF from 127.15.56.1:49849 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":68,"threads_started":1}
Aug 10 23:02:08 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:08 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:08.959204 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:08.953365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:55943)
0810 23:02:08.953747 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:08.953763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:08.953874 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:08.954167 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:08.954171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:08.954421 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:08.954551 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:08.954559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.955438 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.955442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:08.956059 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:08.956068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.956516 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.956520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:08.956602 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:08.957156 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:08.957172 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:08.959068 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":95,"threads_started":1}
W20260810 23:02:08.959338 31397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:08.953508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:55943 (local address 127.15.56.62:44491)
0810 23:02:08.953657 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:08.953662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:08.953841 (+ 179us) server_negotiation.cc:407] Connection header received
0810 23:02:08.953971 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:08.953975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:08.954032 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:08.954122 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:08.954667 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.955322 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.956178 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:08.956412 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:08.956501 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:08.959212 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:55943: BlockingRecv error: recv got EOF from 127.15.56.2:55943 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 23:02:09 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:09 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.3@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:09.147315 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:09.141715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.3:37185)
0810 23:02:09.141994 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:09.142007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:09.142130 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:09.142366 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:09.142368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:09.142560 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:09.142660 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:09.142665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.143421 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.143423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:09.144081 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:09.144093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.144491 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.144493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:09.144554 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:09.145106 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:09.145123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:09.147166 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
W20260810 23:02:09.147418 31397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:09.141816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:37185 (local address 127.15.56.62:44491)
0810 23:02:09.141979 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:09.141982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:09.142074 (+ 92us) server_negotiation.cc:407] Connection header received
0810 23:02:09.142207 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:09.142210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:09.142259 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:09.142340 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:09.142764 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.143305 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.144188 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.144407 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.144477 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:09.147313 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.3:37185: BlockingRecv error: recv got EOF from 127.15.56.3:37185 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 23:02:09 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:09 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.1@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:09.758602 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:09.752872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.1:38899)
0810 23:02:09.753166 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:09.753179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:09.753270 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:09.753561 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:09.753564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:09.753770 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:09.753878 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:09.753884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.754645 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.754647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:09.755239 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:09.755249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.755792 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.755796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:09.755884 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:09.756443 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:09.756458 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:09.758470 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
W20260810 23:02:09.758733 31402 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:09.753018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:38899 (local address 127.15.56.62:44491)
0810 23:02:09.753245 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:09.753249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:09.753263 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:02:09.753359 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:09.753361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:09.753406 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:09.753483 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:09.753990 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.754529 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.755381 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.755650 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.755773 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:09.758611 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.1:38899: BlockingRecv error: recv got EOF from 127.15.56.1:38899 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":63,"threads_started":1}
Aug 10 23:02:09 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:02:09 dist-test-slave-6bbx krb5kdc[29673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.56.2@KRBTEST.COM for oryx/127.15.56.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:02:09.965214 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:09.960134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44491 (local address 127.15.56.2:60295)
0810 23:02:09.960420 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:09.960433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:09.960529 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:09.960738 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:09.960740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:09.960934 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:09.961038 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:09.961043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.961772 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.961775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:09.962359 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:09.962367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.962779 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.962782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:09.962854 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:09.963357 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:09.963368 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:09.965083 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.56.62:44491: Server oryx/127.15.56.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":76,"threads_started":1}
W20260810 23:02:09.965368 31402 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:09.960230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:60295 (local address 127.15.56.62:44491)
0810 23:02:09.960367 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:09.960370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:09.960492 (+ 122us) server_negotiation.cc:407] Connection header received
0810 23:02:09.960599 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:09.960601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:09.960643 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:09.960726 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:09.961131 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.961655 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.962453 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:09.962667 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:09.962755 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:09.965255 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.56.2:60295: BlockingRecv error: recv got EOF from 127.15.56.2:60295 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:02:10.037061 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 30635
I20260810 23:02:10.068239 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 30880
I20260810 23:02:10.076186 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 31094
I20260810 23:02:10.089286 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 30524
2026-08-10T23:02:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18272 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T23:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:02:10Z Disabled control of system clock
I20260810 23:02:10.128273 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:41365
--webserver_interface=127.15.56.62
--webserver_port=0
--builtin_ntp_servers=127.15.56.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:41365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:02:10.240624 31412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:10.240898 31412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:10.240932 31412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:10.244902 31412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:02:10.244956 31412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:10.244971 31412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:10.244985 31412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:02:10.244998 31412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:02:10.249662 31412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:41365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:41365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:10.250711 31412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:10.252084 31412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:02:10.260401 31425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:10.260401 31428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:10.260529 31412 server_base.cc:1034] running on GCE node
W20260810 23:02:10.260712 31426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:10.261286 31412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:10.262389 31412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:10.263573 31412 hybrid_clock.cc:648] HybridClock initialized: now 1786402930263557 us; error 29 us; skew 500 ppm
I20260810 23:02:10.265389 31412 webserver.cc:460] Webserver started at http://127.15.56.62:35761/ using document root <none> and password file <none>
I20260810 23:02:10.265923 31412 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:10.265981 31412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:10.266162 31412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:10.268198 31412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "73fbea660c4244f1b3f559f92087d09e"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "d6c11a9ca0719d29b0a8a173f4207613"
server_key_iv: "7566e880ba87f6a36084078a39745702"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.268711 31412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "73fbea660c4244f1b3f559f92087d09e"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "d6c11a9ca0719d29b0a8a173f4207613"
server_key_iv: "7566e880ba87f6a36084078a39745702"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.272497 31412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:02:10.274940 31435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:10.276238 31412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:02:10.276366 31412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "73fbea660c4244f1b3f559f92087d09e"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "d6c11a9ca0719d29b0a8a173f4207613"
server_key_iv: "7566e880ba87f6a36084078a39745702"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.276470 31412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:10.299171 31412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:10.300055 31412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:10.300261 31412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:10.307688 31412 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:41365
I20260810 23:02:10.307722 31518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:41365 every 8 connection(s)
I20260810 23:02:10.308729 31412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:02:10.312299 31519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:02:10.314586 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 31412
I20260810 23:02:10.314716 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:02:10.314970 15584 external_mini_cluster.cc:1428] Setting key fceb30b68a5bb7039a828b59de0a5c39
I20260810 23:02:10.318681 31519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e: Bootstrap starting.
I20260810 23:02:10.321041 31519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e: Neither blocks nor log segments found. Creating new log.
I20260810 23:02:10.321954 31519 log.cc:824] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:02:10.324506 31519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e: No bootstrap required, opened a new log
I20260810 23:02:10.325126 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:10.316509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.15.56.62:41365)
0810 23:02:10.317258 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:10.317274 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:10.317315 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:02:10.318004 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:10.318025 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:10.318473 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:10.318809 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:10.319258 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.320302 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.321221 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.321939 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.322125 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:10.322958 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:10.322965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:10.322982 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:02:10.323472 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:10.323544 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:02:10.323572 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:10.323728 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:10.323844 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 23:02:10.324127 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":145,"threads_started":1}
I20260810 23:02:10.328729 31519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fbea660c4244f1b3f559f92087d09e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41365 } }
I20260810 23:02:10.328953 31519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:02:10.328994 31519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73fbea660c4244f1b3f559f92087d09e, State: Initialized, Role: FOLLOWER
I20260810 23:02:10.329366 31519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fbea660c4244f1b3f559f92087d09e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41365 } }
I20260810 23:02:10.329483 31519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:02:10.329528 31519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:02:10.329597 31519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:02:10.330469 31519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fbea660c4244f1b3f559f92087d09e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41365 } }
I20260810 23:02:10.330771 31519 leader_election.cc:302] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73fbea660c4244f1b3f559f92087d09e; no voters:
I20260810 23:02:10.331069 31519 leader_election.cc:288] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:02:10.331164 31529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:02:10.331375 31529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [term 1 LEADER]: Becoming Leader. State: Replica: 73fbea660c4244f1b3f559f92087d09e, State: Running, Role: LEADER
I20260810 23:02:10.331753 31529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fbea660c4244f1b3f559f92087d09e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41365 } }
I20260810 23:02:10.332314 31519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:02:10.333561 31530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73fbea660c4244f1b3f559f92087d09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fbea660c4244f1b3f559f92087d09e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41365 } } }
I20260810 23:02:10.333611 31532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73fbea660c4244f1b3f559f92087d09e. Latest consensus state: current_term: 1 leader_uuid: "73fbea660c4244f1b3f559f92087d09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73fbea660c4244f1b3f559f92087d09e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41365 } } }
I20260810 23:02:10.333705 31530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [sys.catalog]: This master's current role is: LEADER
I20260810 23:02:10.333722 31532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e [sys.catalog]: This master's current role is: LEADER
I20260810 23:02:10.334138 31542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:02:10.337126 31542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:02:10.342394 31542 catalog_manager.cc:1347] Generated new cluster ID: f3b68b9b28a84266ae93ba21d44ff7b9
I20260810 23:02:10.342468 31542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:02:10.364048 31542 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:02:10.365036 31542 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:02:10.375344 31542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73fbea660c4244f1b3f559f92087d09e: Generated new TSK 0
I20260810 23:02:10.376138 31542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:02:10.384159 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41365
--builtin_ntp_servers=127.15.56.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 23:02:10.502709 31565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:10.503036 31565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:10.503093 31565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:10.507316 31565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:10.507380 31565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:10.507454 31565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:02:10.512328 31565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:10.513429 31565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:10.514899 31565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:02:10.524670 31574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:10.524673 31578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:10.524886 31565 server_base.cc:1034] running on GCE node
W20260810 23:02:10.524673 31572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:10.525413 31565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:10.526014 31565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:10.527169 31565 hybrid_clock.cc:648] HybridClock initialized: now 1786402930527151 us; error 32 us; skew 500 ppm
I20260810 23:02:10.529094 31565 webserver.cc:460] Webserver started at http://127.15.56.1:46065/ using document root <none> and password file <none>
I20260810 23:02:10.529670 31565 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:10.529736 31565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:10.529935 31565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:10.532100 31565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "c66bb963d9e54766a60a1afd589d7c32"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "0503815dc5ed2264284a8bbea858ba96"
server_key_iv: "4987235074b31b6a87cc9e1f6ef61fb6"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.532688 31565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "c66bb963d9e54766a60a1afd589d7c32"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "0503815dc5ed2264284a8bbea858ba96"
server_key_iv: "4987235074b31b6a87cc9e1f6ef61fb6"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.536547 31565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:02:10.539057 31588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:10.540306 31565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:02:10.540421 31565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "c66bb963d9e54766a60a1afd589d7c32"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "0503815dc5ed2264284a8bbea858ba96"
server_key_iv: "4987235074b31b6a87cc9e1f6ef61fb6"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.540521 31565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:10.573201 31565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:10.573988 31565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:10.574177 31565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:10.574817 31565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:02:10.575856 31565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:02:10.575906 31565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:10.575951 31565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:02:10.575981 31565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:10.585021 31565 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:37627
I20260810 23:02:10.585065 31768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:37627 every 8 connection(s)
I20260810 23:02:10.585942 31565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:02:10.592439 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 31565
I20260810 23:02:10.592563 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:02:10.592876 15584 external_mini_cluster.cc:1428] Setting key 2f29ab77efc7084e0260a194827290bc
I20260810 23:02:10.594971 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41365
--builtin_ntp_servers=127.15.56.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 23:02:10.595629 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:10.587775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:47217 (local address 127.15.56.62:41365)
0810 23:02:10.587929 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:10.587934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:10.589087 (+ 1153us) server_negotiation.cc:407] Connection header received
0810 23:02:10.590457 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:10.590460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:10.590509 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:10.590589 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:10.591466 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.592062 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.593587 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.593875 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.593973 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:10.595152 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:10.595156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:10.595158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:02:10.595170 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:10.595193 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:02:10.595203 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:10.595263 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:10.595425 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 23:02:10.595487 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:02:10.596467 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:10.588253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41365 (local address 127.15.56.1:47217)
0810 23:02:10.588942 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:10.588983 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:10.590241 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:10.590749 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:10.590756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:10.590854 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:10.591343 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:10.591356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.592169 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.592173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:10.593469 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:10.593480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.594009 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.594012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:10.594203 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:10.594807 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:02:10.594809 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:10.594869 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:02:10.594951 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:02:10.595297 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:02:10.595306 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:02:10.595311 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:02:10.595408 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:02:10.595625 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
I20260810 23:02:10.597609 31769 heartbeater.cc:347] Connected to a master server at 127.15.56.62:41365
I20260810 23:02:10.597864 31769 heartbeater.cc:464] Registering TS with master...
I20260810 23:02:10.598459 31769 heartbeater.cc:511] Master 127.15.56.62:41365 requested a full tablet report, sending...
I20260810 23:02:10.600164 31459 ts_manager.cc:194] Registered new tserver with Master: c66bb963d9e54766a60a1afd589d7c32 (127.15.56.1:37627)
I20260810 23:02:10.601742 31459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.1:47217
W20260810 23:02:10.711560 31772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:10.711836 31772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:10.711874 31772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:10.715757 31772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:10.715811 31772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:10.715862 31772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:02:10.720404 31772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:10.721383 31772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:10.722765 31772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:02:10.733134 31783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:10.733134 31785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:10.733166 31781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:10.733507 31772 server_base.cc:1034] running on GCE node
I20260810 23:02:10.733978 31772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:10.734663 31772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:10.735828 31772 hybrid_clock.cc:648] HybridClock initialized: now 1786402930735808 us; error 33 us; skew 500 ppm
I20260810 23:02:10.737810 31772 webserver.cc:460] Webserver started at http://127.15.56.2:43519/ using document root <none> and password file <none>
I20260810 23:02:10.738400 31772 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:10.738466 31772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:10.738683 31772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:10.741000 31772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "900ceb0b079c4282baa01844358c8942"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "d75f114156121c46633a75cf7958ae3f"
server_key_iv: "93015b8f81c5edc465da78c44050da32"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.741555 31772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "900ceb0b079c4282baa01844358c8942"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "d75f114156121c46633a75cf7958ae3f"
server_key_iv: "93015b8f81c5edc465da78c44050da32"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.745429 31772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:02:10.748203 31794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:10.749511 31772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:02:10.749636 31772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "900ceb0b079c4282baa01844358c8942"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "d75f114156121c46633a75cf7958ae3f"
server_key_iv: "93015b8f81c5edc465da78c44050da32"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.749737 31772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:10.773655 31772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:10.774425 31772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:10.774608 31772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:10.775234 31772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:02:10.776249 31772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:02:10.776299 31772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:10.776355 31772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:02:10.776382 31772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:10.785187 31772 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:46783
I20260810 23:02:10.785226 31968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:46783 every 8 connection(s)
I20260810 23:02:10.786141 31772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:02:10.792693 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 31772
I20260810 23:02:10.792850 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:02:10.793169 15584 external_mini_cluster.cc:1428] Setting key fd753b6b7c38366c49105fe553728415
I20260810 23:02:10.795509 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41365
--builtin_ntp_servers=127.15.56.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 23:02:10.796239 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:10.788164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:36597 (local address 127.15.56.62:41365)
0810 23:02:10.788301 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:10.788305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:10.789378 (+ 1073us) server_negotiation.cc:407] Connection header received
0810 23:02:10.790751 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:10.790754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:10.790805 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:10.790880 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:10.791831 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.792366 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.793950 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.794261 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.794370 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:10.795706 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:10.795711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:10.795715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:02:10.795734 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:10.795764 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:02:10.795776 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:10.795857 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:10.796026 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 23:02:10.796085 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":48}
I20260810 23:02:10.797498 31971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:10.788529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41365 (local address 127.15.56.2:36597)
0810 23:02:10.789211 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:10.789254 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:10.790539 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:10.791041 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:10.791048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:10.791135 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:10.791697 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:10.791717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.792499 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.792503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:10.793821 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:10.793834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.794428 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.794431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:10.794609 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:10.795273 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:02:10.795275 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:10.795349 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:02:10.795451 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:02:10.795866 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:02:10.795877 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:02:10.795884 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:02:10.796026 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 23:02:10.796379 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260810 23:02:10.798720 31969 heartbeater.cc:347] Connected to a master server at 127.15.56.62:41365
I20260810 23:02:10.798971 31969 heartbeater.cc:464] Registering TS with master...
I20260810 23:02:10.799544 31969 heartbeater.cc:511] Master 127.15.56.62:41365 requested a full tablet report, sending...
I20260810 23:02:10.800700 31459 ts_manager.cc:194] Registered new tserver with Master: 900ceb0b079c4282baa01844358c8942 (127.15.56.2:46783)
I20260810 23:02:10.801487 31459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.2:36597
W20260810 23:02:10.915136 31973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:10.915527 31973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:10.915568 31973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:10.919694 31973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:10.919766 31973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:10.919823 31973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:02:10.924788 31973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:10.925916 31973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:10.927356 31973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:02:10.937927 31982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:10.937939 31980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:10.937976 31973 server_base.cc:1034] running on GCE node
W20260810 23:02:10.937927 31979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:10.938589 31973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:10.939309 31973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:10.940488 31973 hybrid_clock.cc:648] HybridClock initialized: now 1786402930940466 us; error 34 us; skew 500 ppm
I20260810 23:02:10.942737 31973 webserver.cc:460] Webserver started at http://127.15.56.3:38141/ using document root <none> and password file <none>
I20260810 23:02:10.943363 31973 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:10.943439 31973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:10.943648 31973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:10.945932 31973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "85a5f1e6af4f4388814404f6241ac15c"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "2598a8df412bcccbcfc30a21da374bc5"
server_key_iv: "a3987d8f84eb881b5816d7b3cf692486"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.946462 31973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "85a5f1e6af4f4388814404f6241ac15c"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "2598a8df412bcccbcfc30a21da374bc5"
server_key_iv: "a3987d8f84eb881b5816d7b3cf692486"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.950233 31973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:02:10.953025 31990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:10.954468 31973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:02:10.954595 31973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "85a5f1e6af4f4388814404f6241ac15c"
format_stamp: "Formatted at 2026-08-10 23:02:10 on dist-test-slave-6bbx"
server_key: "2598a8df412bcccbcfc30a21da374bc5"
server_key_iv: "a3987d8f84eb881b5816d7b3cf692486"
server_key_version: "encryptionkey@0"
I20260810 23:02:10.954712 31973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:10.971603 31973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:10.972450 31973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:10.972617 31973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:10.973208 31973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:02:10.974226 31973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:02:10.974277 31973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:10.974342 31973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:02:10.974368 31973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:10.983388 31973 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:39527
I20260810 23:02:10.983433 32152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:39527 every 8 connection(s)
I20260810 23:02:10.984447 31973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:02:10.993897 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 31973
I20260810 23:02:10.994010 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:02:10.994278 15584 external_mini_cluster.cc:1428] Setting key 0fb282f56b01e6e1e5e9200bf01d61ef
I20260810 23:02:10.995154 31527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:10.986747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:47339 (local address 127.15.56.62:41365)
0810 23:02:10.986964 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:10.986969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:10.988283 (+ 1314us) server_negotiation.cc:407] Connection header received
0810 23:02:10.989794 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:10.989798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:10.989864 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:10.989952 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:10.990990 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.991552 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.992951 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.993204 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.993302 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:10.994645 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:10.994649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:10.994651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:02:10.994665 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:10.994704 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:02:10.994712 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:10.994770 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:10.994936 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:02:10.995008 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 23:02:10.996260 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:10.987267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41365 (local address 127.15.56.3:47339)
0810 23:02:10.988108 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:10.988161 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:10.989536 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:10.990137 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:10.990146 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:10.990244 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:10.990815 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:10.990829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.991775 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.991779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:10.992791 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:10.992803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:10.993389 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:10.993392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:10.993580 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:10.994321 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:02:10.994323 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:10.994377 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:02:10.994474 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:02:10.994791 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:02:10.994802 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:02:10.994808 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:02:10.994927 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 23:02:10.995280 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":147,"threads_started":1}
I20260810 23:02:10.997378 32153 heartbeater.cc:347] Connected to a master server at 127.15.56.62:41365
I20260810 23:02:10.997614 32153 heartbeater.cc:464] Registering TS with master...
I20260810 23:02:10.998200 32153 heartbeater.cc:511] Master 127.15.56.62:41365 requested a full tablet report, sending...
I20260810 23:02:10.999226 31459 ts_manager.cc:194] Registered new tserver with Master: 85a5f1e6af4f4388814404f6241ac15c (127.15.56.3:39527)
I20260810 23:02:10.999835 31459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.3:47339
I20260810 23:02:11.008745 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:02:11.011747 31527 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:11.010714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.15.56.62:41365)
0810 23:02:11.010889 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:11.010894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:11.011027 (+ 133us) server_negotiation.cc:407] Connection header received
0810 23:02:11.011164 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:11.011167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:11.011211 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:11.011277 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:11.011585 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55100: BlockingRecv error: recv got EOF from 127.0.0.1:55100 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:02:11.012449 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 31565
I20260810 23:02:11.026017 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 31772
I20260810 23:02:11.055177 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 31973
I20260810 23:02:11.090296 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 31412
2026-08-10T23:02:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (998 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T23:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:02:11Z Disabled control of system clock
I20260810 23:02:11.126945 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:37453
--webserver_interface=127.15.56.62
--webserver_port=0
--builtin_ntp_servers=127.15.56.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:37453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:02:11.242223 32170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:11.242545 32170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:11.242590 32170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:11.246729 32170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:02:11.246804 32170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:11.246819 32170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:11.246834 32170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:02:11.246848 32170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:02:11.251806 32170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:37453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:37453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:11.253033 32170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:11.254477 32170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:02:11.263240 32178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:11.263274 32170 server_base.cc:1034] running on GCE node
W20260810 23:02:11.263266 32182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:11.263469 32177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:11.264092 32170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:11.265192 32170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:11.266358 32170 hybrid_clock.cc:648] HybridClock initialized: now 1786402931266327 us; error 44 us; skew 500 ppm
I20260810 23:02:11.268316 32170 webserver.cc:460] Webserver started at http://127.15.56.62:34471/ using document root <none> and password file <none>
I20260810 23:02:11.268899 32170 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:11.268965 32170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:11.269163 32170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:11.271246 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "285117dd4b7645b99330596d175239a7"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "9766825b29f41e9e9e8a730fb2d75737"
server_key_iv: "91d2b99c95734c2efa2b1f2dd2c5c684"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.271857 32170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "285117dd4b7645b99330596d175239a7"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "9766825b29f41e9e9e8a730fb2d75737"
server_key_iv: "91d2b99c95734c2efa2b1f2dd2c5c684"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.275612 32170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:02:11.278059 32195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:11.279335 32170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:02:11.279454 32170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "285117dd4b7645b99330596d175239a7"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "9766825b29f41e9e9e8a730fb2d75737"
server_key_iv: "91d2b99c95734c2efa2b1f2dd2c5c684"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.279556 32170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:11.284176 32170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:11.284787 32170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:11.284948 32170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:11.291531 32170 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:37453
I20260810 23:02:11.291553 32274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:37453 every 8 connection(s)
I20260810 23:02:11.292521 32170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:02:11.293651 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 32170
I20260810 23:02:11.293761 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:02:11.294032 15584 external_mini_cluster.cc:1428] Setting key bd4ca87103de34b4b4a0592598fd7d1d
I20260810 23:02:11.296325 32276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:02:11.299808 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:11.295465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.15.56.62:37453)
0810 23:02:11.296183 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:11.296200 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:11.296238 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:02:11.296926 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:11.296945 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:11.297347 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:11.297680 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:11.298499 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:11.298504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:11.298516 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:02:11.298526 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:11.298585 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:02:11.298601 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:11.298732 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:11.298812 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 23:02:11.299050 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":192,"threads_started":1}
I20260810 23:02:11.302724 32276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7: Bootstrap starting.
I20260810 23:02:11.305013 32276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7: Neither blocks nor log segments found. Creating new log.
I20260810 23:02:11.305773 32276 log.cc:824] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:02:11.307982 32276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7: No bootstrap required, opened a new log
I20260810 23:02:11.311539 32276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285117dd4b7645b99330596d175239a7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37453 } }
I20260810 23:02:11.311826 32276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:02:11.311877 32276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285117dd4b7645b99330596d175239a7, State: Initialized, Role: FOLLOWER
I20260810 23:02:11.312286 32276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285117dd4b7645b99330596d175239a7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37453 } }
I20260810 23:02:11.312413 32276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:02:11.312469 32276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:02:11.312556 32276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:02:11.313434 32276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285117dd4b7645b99330596d175239a7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37453 } }
I20260810 23:02:11.313798 32276 leader_election.cc:302] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 285117dd4b7645b99330596d175239a7; no voters:
I20260810 23:02:11.314137 32276 leader_election.cc:288] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:02:11.314261 32284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:02:11.314442 32284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [term 1 LEADER]: Becoming Leader. State: Replica: 285117dd4b7645b99330596d175239a7, State: Running, Role: LEADER
I20260810 23:02:11.314803 32284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285117dd4b7645b99330596d175239a7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37453 } }
I20260810 23:02:11.315546 32276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:02:11.316728 32289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "285117dd4b7645b99330596d175239a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285117dd4b7645b99330596d175239a7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37453 } } }
I20260810 23:02:11.316870 32289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:02:11.317186 32288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 285117dd4b7645b99330596d175239a7. Latest consensus state: current_term: 1 leader_uuid: "285117dd4b7645b99330596d175239a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285117dd4b7645b99330596d175239a7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37453 } } }
I20260810 23:02:11.317322 32288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:02:11.317363 32299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:02:11.320681 32299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:02:11.326835 32299 catalog_manager.cc:1347] Generated new cluster ID: ac719c06309a4836a8bb4ecdee24c09f
I20260810 23:02:11.326913 32299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:02:11.344069 32299 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:02:11.344503 32299 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:02:11.353009 32299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 285117dd4b7645b99330596d175239a7: Generated new TSK 0
I20260810 23:02:11.353829 32299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:02:11.357941 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:37453
--builtin_ntp_servers=127.15.56.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:02:11.487949 32315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:11.488346 32315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:11.488394 32315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:11.492831 32315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:11.492923 32315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:11.492985 32315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:02:11.498224 32315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:37453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:11.499449 32315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:11.501047 32315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:02:11.511966 32323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:11.511952 32321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:11.512001 32315 server_base.cc:1034] running on GCE node
W20260810 23:02:11.511952 32320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:11.512705 32315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:11.513386 32315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:11.514544 32315 hybrid_clock.cc:648] HybridClock initialized: now 1786402931514515 us; error 39 us; skew 500 ppm
I20260810 23:02:11.516696 32315 webserver.cc:460] Webserver started at http://127.15.56.1:34889/ using document root <none> and password file <none>
I20260810 23:02:11.517352 32315 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:11.517422 32315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:11.517645 32315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:11.519927 32315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "e4320c9919f44ddcaaf30a823bb0333f"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "f44326cad4331134e502c6f8b570ed29"
server_key_iv: "5e8bfd0ee23ef36240a82aed2f7290ba"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.520533 32315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "e4320c9919f44ddcaaf30a823bb0333f"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "f44326cad4331134e502c6f8b570ed29"
server_key_iv: "5e8bfd0ee23ef36240a82aed2f7290ba"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.524662 32315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:02:11.527603 32331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:11.529067 32315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:02:11.529255 32315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "e4320c9919f44ddcaaf30a823bb0333f"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "f44326cad4331134e502c6f8b570ed29"
server_key_iv: "5e8bfd0ee23ef36240a82aed2f7290ba"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.529379 32315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:11.534266 32315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:11.534946 32315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:11.535117 32315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:11.535751 32315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:02:11.536774 32315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:02:11.536824 32315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:11.536876 32315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:02:11.536904 32315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:11.545635 32315 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:42839
I20260810 23:02:11.545662 32493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:42839 every 8 connection(s)
I20260810 23:02:11.546622 32315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:02:11.546814 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 32315
I20260810 23:02:11.546913 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:02:11.547247 15584 external_mini_cluster.cc:1428] Setting key de690ce0fe193b1ecf28ecd29f5ac703
I20260810 23:02:11.549593 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:37453
--builtin_ntp_servers=127.15.56.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 23:02:11.554858 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:11.548723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:41879 (local address 127.15.56.62:37453)
0810 23:02:11.548852 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:11.548857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:11.550416 (+ 1559us) server_negotiation.cc:407] Connection header received
0810 23:02:11.552334 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:11.552338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:11.552388 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:11.552457 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:11.554094 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:11.554098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:11.554101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:02:11.554119 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:11.554149 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:02:11.554161 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:11.554285 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:11.554585 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 23:02:11.554664 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:02:11.555819 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:11.549349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:37453 (local address 127.15.56.1:41879)
0810 23:02:11.550251 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:11.550300 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:11.551940 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:11.552695 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:11.552707 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:11.553004 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:11.553729 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:02:11.553731 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:11.553800 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:02:11.553898 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:02:11.554310 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:02:11.554321 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:02:11.554327 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:02:11.554520 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 23:02:11.554800 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":163,"threads_started":1}
I20260810 23:02:11.557497 32495 heartbeater.cc:347] Connected to a master server at 127.15.56.62:37453
I20260810 23:02:11.557816 32495 heartbeater.cc:464] Registering TS with master...
I20260810 23:02:11.558318 32495 heartbeater.cc:511] Master 127.15.56.62:37453 requested a full tablet report, sending...
I20260810 23:02:11.560245 32223 ts_manager.cc:194] Registered new tserver with Master: e4320c9919f44ddcaaf30a823bb0333f (127.15.56.1:42839)
W20260810 23:02:11.672962 32499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:11.673246 32499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:11.673293 32499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:11.677330 32499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:11.677394 32499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:11.677450 32499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:02:11.682415 32499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:37453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:11.683796 32499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:11.685386 32499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:02:11.695366 32509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:11.695370 32513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:11.695402 32507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:11.695629 32499 server_base.cc:1034] running on GCE node
I20260810 23:02:11.696175 32499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:11.696815 32499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:11.697973 32499 hybrid_clock.cc:648] HybridClock initialized: now 1786402931697953 us; error 30 us; skew 500 ppm
I20260810 23:02:11.699877 32499 webserver.cc:460] Webserver started at http://127.15.56.2:33121/ using document root <none> and password file <none>
I20260810 23:02:11.700464 32499 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:11.700533 32499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:11.700743 32499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:11.702837 32499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "2ef6471b39b74d52adfa8343e7f5d6ea"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "845971a19ab072e50e07342aa6f5a460"
server_key_iv: "67d884e87c85a78ce4e84520165dd04f"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.703389 32499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "2ef6471b39b74d52adfa8343e7f5d6ea"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "845971a19ab072e50e07342aa6f5a460"
server_key_iv: "67d884e87c85a78ce4e84520165dd04f"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.707109 32499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:02:11.709962 32522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:11.711287 32499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:02:11.711416 32499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "2ef6471b39b74d52adfa8343e7f5d6ea"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "845971a19ab072e50e07342aa6f5a460"
server_key_iv: "67d884e87c85a78ce4e84520165dd04f"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.711534 32499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:11.716954 32499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:11.717693 32499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:11.717873 32499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:11.718508 32499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:02:11.719511 32499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:02:11.719573 32499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:11.719630 32499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:02:11.719655 32499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:11.728345 32499 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:33373
I20260810 23:02:11.728430 32688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:33373 every 8 connection(s)
I20260810 23:02:11.729385 32499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:02:11.735992 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:11.731267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:52401 (local address 127.15.56.62:37453)
0810 23:02:11.731401 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:11.731405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:11.732681 (+ 1276us) server_negotiation.cc:407] Connection header received
0810 23:02:11.734158 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:11.734162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:11.734217 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:11.734301 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:11.735494 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:11.735497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:11.735499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:02:11.735513 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:11.735536 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:02:11.735545 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:11.735631 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:11.735787 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 23:02:11.735841 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:02:11.736945 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:11.731710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:37453 (local address 127.15.56.2:52401)
0810 23:02:11.732459 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:11.732571 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:11.733907 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:11.734497 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:11.734505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:11.734605 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:11.735189 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:02:11.735191 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:11.735239 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:02:11.735318 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:02:11.735618 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:02:11.735627 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:02:11.735633 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:02:11.735842 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 23:02:11.736121 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":126,"threads_started":1}
I20260810 23:02:11.738168 32689 heartbeater.cc:347] Connected to a master server at 127.15.56.62:37453
I20260810 23:02:11.738416 32689 heartbeater.cc:464] Registering TS with master...
I20260810 23:02:11.738862 32689 heartbeater.cc:511] Master 127.15.56.62:37453 requested a full tablet report, sending...
I20260810 23:02:11.739005 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 32499
I20260810 23:02:11.739112 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:02:11.739399 15584 external_mini_cluster.cc:1428] Setting key ae735b8bb09a58cf242d1e008cdf8e4a
I20260810 23:02:11.740154 32223 ts_manager.cc:194] Registered new tserver with Master: 2ef6471b39b74d52adfa8343e7f5d6ea (127.15.56.2:33373)
I20260810 23:02:11.741888 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:37453
--builtin_ntp_servers=127.15.56.20:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:02:11.862057 32694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:11.862350 32694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:11.862387 32694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:11.866426 32694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:11.866488 32694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:11.866544 32694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:02:11.871305 32694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:37453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:11.872455 32694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:11.873826 32694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:02:11.883409 32707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:11.883404 32704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:11.883431 32705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:11.883777 32694 server_base.cc:1034] running on GCE node
I20260810 23:02:11.884241 32694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:11.884891 32694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:11.886080 32694 hybrid_clock.cc:648] HybridClock initialized: now 1786402931886064 us; error 25 us; skew 500 ppm
I20260810 23:02:11.888118 32694 webserver.cc:460] Webserver started at http://127.15.56.3:37979/ using document root <none> and password file <none>
I20260810 23:02:11.888720 32694 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:11.888790 32694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:11.888972 32694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:11.891508 32694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "84d18fba68984cbdb36fd81b801b6b9b"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "c75143294ab0f31e89fa5a4d62089c4f"
server_key_iv: "cd60d90a4c07f991a2a106a5dcf26a24"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.892302 32694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "84d18fba68984cbdb36fd81b801b6b9b"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "c75143294ab0f31e89fa5a4d62089c4f"
server_key_iv: "cd60d90a4c07f991a2a106a5dcf26a24"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.896414 32694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:02:11.899327 32716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:11.900909 32694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:02:11.901059 32694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "84d18fba68984cbdb36fd81b801b6b9b"
format_stamp: "Formatted at 2026-08-10 23:02:11 on dist-test-slave-6bbx"
server_key: "c75143294ab0f31e89fa5a4d62089c4f"
server_key_iv: "cd60d90a4c07f991a2a106a5dcf26a24"
server_key_version: "encryptionkey@0"
I20260810 23:02:11.901202 32694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:11.906085 32694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:11.906715 32694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:11.906877 32694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:11.907434 32694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:02:11.908533 32694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:02:11.908582 32694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:11.908633 32694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:02:11.908659 32694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:11.917099 32694 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:36581
I20260810 23:02:11.917177 404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:36581 every 8 connection(s)
I20260810 23:02:11.918025 32694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:02:11.919855 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 32694
I20260810 23:02:11.919974 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:02:11.920251 15584 external_mini_cluster.cc:1428] Setting key ed7b6903609ad934a3d070674822b665
I20260810 23:02:11.924345 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:11.919898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:37409 (local address 127.15.56.62:37453)
0810 23:02:11.920050 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:11.920054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:11.921278 (+ 1224us) server_negotiation.cc:407] Connection header received
0810 23:02:11.922808 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:11.922810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:11.922852 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:11.922926 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:11.923971 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:11.923974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:11.923976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:02:11.923988 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:11.924008 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:02:11.924016 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:11.924086 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:11.924181 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 23:02:11.924217 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:02:11.925227 408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:11.920351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:37453 (local address 127.15.56.3:37409)
0810 23:02:11.921135 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:11.921192 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:11.922584 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:11.923076 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:11.923083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:11.923176 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:11.923713 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:02:11.923715 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:11.923767 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:02:11.923840 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:02:11.924079 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:02:11.924084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:02:11.924087 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:02:11.924214 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 23:02:11.924419 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":136,"threads_started":1}
I20260810 23:02:11.926271 406 heartbeater.cc:347] Connected to a master server at 127.15.56.62:37453
I20260810 23:02:11.926517 406 heartbeater.cc:464] Registering TS with master...
I20260810 23:02:11.926931 406 heartbeater.cc:511] Master 127.15.56.62:37453 requested a full tablet report, sending...
I20260810 23:02:11.927969 32222 ts_manager.cc:194] Registered new tserver with Master: 84d18fba68984cbdb36fd81b801b6b9b (127.15.56.3:36581)
I20260810 23:02:11.934062 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:02:11.937016 32282 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:02:11.936119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.15.56.62:37453)
0810 23:02:11.936256 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:11.936260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:11.936448 (+ 188us) server_negotiation.cc:407] Connection header received
0810 23:02:11.936565 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:11.936567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:11.936603 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:11.936659 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:11.936881 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41918: BlockingRecv error: recv got EOF from 127.0.0.1:41918 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:02:11.937172 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 32315
I20260810 23:02:11.957880 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 32499
I20260810 23:02:11.972615 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 32694
I20260810 23:02:11.985911 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 32170
2026-08-10T23:02:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (895 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:02:12 dist-test-slave-6bbx krb5kdc[418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:02:12 dist-test-slave-6bbx krb5kdc[418](info): set up 2 sockets
Aug 10 23:02:12 dist-test-slave-6bbx krb5kdc[418](info): commencing operation
krb5kdc: starting...
W20260810 23:02:13.170346 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:02:13 dist-test-slave-6bbx krb5kdc[418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:02:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:02:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:02:13.302779 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43641
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:43641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:02:13.423949 438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:13.424270 438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:13.424331 438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:13.428483 438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:02:13.428557 438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:13.428586 438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:13.428606 438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:02:13.428632 438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:02:13.434388 438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:43641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:13.435653 438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:13.436854 438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:02:13.446440 438 server_base.cc:1034] running on GCE node
W20260810 23:02:13.446359 449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:13.446355 447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:13.446374 452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:13.447293 438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:13.448583 438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:13.449754 438 hybrid_clock.cc:648] HybridClock initialized: now 1786402933449731 us; error 36 us; skew 500 ppm
Aug 10 23:02:13 dist-test-slave-6bbx krb5kdc[418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402933, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:02:13.452652 438 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:02:13.453637 438 webserver.cc:460] Webserver started at http://127.15.56.62:43721/ using document root <none> and password file <none>
I20260810 23:02:13.454206 438 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:13.454276 438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:13.454483 438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:13.456700 438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "c3666b81e221474090795f578ae80862"
format_stamp: "Formatted at 2026-08-10 23:02:13 on dist-test-slave-6bbx"
server_key: "eac1be2bc1b74697a88c3f2137867e34"
server_key_iv: "6a516912a901d77ccb09977b73472ad9"
server_key_version: "encryptionkey@0"
I20260810 23:02:13.457283 438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "c3666b81e221474090795f578ae80862"
format_stamp: "Formatted at 2026-08-10 23:02:13 on dist-test-slave-6bbx"
server_key: "eac1be2bc1b74697a88c3f2137867e34"
server_key_iv: "6a516912a901d77ccb09977b73472ad9"
server_key_version: "encryptionkey@0"
I20260810 23:02:13.461308 438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:02:13.463907 462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:13.465323 438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:02:13.465463 438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "c3666b81e221474090795f578ae80862"
format_stamp: "Formatted at 2026-08-10 23:02:13 on dist-test-slave-6bbx"
server_key: "eac1be2bc1b74697a88c3f2137867e34"
server_key_iv: "6a516912a901d77ccb09977b73472ad9"
server_key_version: "encryptionkey@0"
I20260810 23:02:13.465579 438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:13.478933 438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:13.482436 438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:13.482641 438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:13.489966 438 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:43641
I20260810 23:02:13.490002 532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:43641 every 8 connection(s)
I20260810 23:02:13.490983 438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:02:13.493968 533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:02:13.500001 533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862: Bootstrap starting.
I20260810 23:02:13.500950 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 438
I20260810 23:02:13.501052 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:02:13.501335 15584 external_mini_cluster.cc:1428] Setting key c0eb9401eb9d6cbd82a6150b1dac541e
I20260810 23:02:13.502882 533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862: Neither blocks nor log segments found. Creating new log.
I20260810 23:02:13.503645 533 log.cc:824] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862: Log is configured to *not* fsync() on all Append() calls
I20260810 23:02:13.505589 533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862: No bootstrap required, opened a new log
I20260810 23:02:13.509191 533 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3666b81e221474090795f578ae80862" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43641 } }
I20260810 23:02:13.509439 533 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:02:13.509502 533 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3666b81e221474090795f578ae80862, State: Initialized, Role: FOLLOWER
I20260810 23:02:13.509936 533 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3666b81e221474090795f578ae80862" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43641 } }
Aug 10 23:02:13 dist-test-slave-6bbx krb5kdc[418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:02:13.510070 533 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:02:13.510133 533 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:02:13.510232 533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:02:13.511169 533 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3666b81e221474090795f578ae80862" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43641 } }
I20260810 23:02:13.511525 533 leader_election.cc:302] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3666b81e221474090795f578ae80862; no voters:
I20260810 23:02:13.511942 533 leader_election.cc:288] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:02:13.512023 538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:02:13.512226 538 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [term 1 LEADER]: Becoming Leader. State: Replica: c3666b81e221474090795f578ae80862, State: Running, Role: LEADER
I20260810 23:02:13.512603 538 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3666b81e221474090795f578ae80862" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43641 } }
I20260810 23:02:13.513358 533 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:02:13.514442 543 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3666b81e221474090795f578ae80862. Latest consensus state: current_term: 1 leader_uuid: "c3666b81e221474090795f578ae80862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3666b81e221474090795f578ae80862" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43641 } } }
I20260810 23:02:13.514580 543 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [sys.catalog]: This master's current role is: LEADER
I20260810 23:02:13.514969 553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:02:13.514969 541 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3666b81e221474090795f578ae80862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3666b81e221474090795f578ae80862" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43641 } } }
I20260810 23:02:13.515242 541 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862 [sys.catalog]: This master's current role is: LEADER
I20260810 23:02:13.516655 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:13.502535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.15.56.62:43641)
0810 23:02:13.503013 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:13.503030 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:13.503069 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:02:13.503822 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:13.503845 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:13.504204 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:13.504523 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:13.505335 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:13.506375 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:13.507339 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:13.507917 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:13.508087 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:13.510673 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:13.510679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:13.510693 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:02:13.510722 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:13.512715 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:02:13.513225 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:02:13.513232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:02:13.513238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:02:13.513377 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:02:13.513664 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:02:13.513668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:02:13.513669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:02:13.514249 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:13.514460 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:13.515178 (+ 718us) server_negotiation.cc:299] Negotiation successful
0810 23:02:13.515544 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
I20260810 23:02:13.518493 553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:02:13.524503 553 catalog_manager.cc:1347] Generated new cluster ID: d62013d070a24901b697f12d6ea70c97
I20260810 23:02:13.524593 553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:02:13.533919 553 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:02:13.535171 553 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:02:13.541951 553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862: Generated new TSK 0
I20260810 23:02:13.542862 553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:02:13.574467 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43641
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:02:13.697316 573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:13.697635 573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:13.697683 573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:13.701807 573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:13.701875 573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:13.701934 573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:02:13.707437 573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:13.708588 573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:13.709664 573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:02:13.720127 587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:13.720127 585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:13.720129 592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:13.720173 573 server_base.cc:1034] running on GCE node
I20260810 23:02:13.720870 573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:13.721657 573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:13.722862 573 hybrid_clock.cc:648] HybridClock initialized: now 1786402933722836 us; error 34 us; skew 500 ppm
Aug 10 23:02:13 dist-test-slave-6bbx krb5kdc[418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402933, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:02:13.725873 573 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:02:13.726892 573 webserver.cc:460] Webserver started at http://127.15.56.1:45197/ using document root <none> and password file <none>
I20260810 23:02:13.727479 573 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:13.727542 573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:13.727795 573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:13.730050 573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "2da8daec96144034918df64101092513"
format_stamp: "Formatted at 2026-08-10 23:02:13 on dist-test-slave-6bbx"
server_key: "aa6c7444d3bfd8bb0d3b1ee2a59c666b"
server_key_iv: "014b30f6eac670cac6a48b2141220084"
server_key_version: "encryptionkey@0"
I20260810 23:02:13.730649 573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "2da8daec96144034918df64101092513"
format_stamp: "Formatted at 2026-08-10 23:02:13 on dist-test-slave-6bbx"
server_key: "aa6c7444d3bfd8bb0d3b1ee2a59c666b"
server_key_iv: "014b30f6eac670cac6a48b2141220084"
server_key_version: "encryptionkey@0"
I20260810 23:02:13.734966 573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 23:02:13.737629 601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:13.738965 573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:02:13.739079 573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "2da8daec96144034918df64101092513"
format_stamp: "Formatted at 2026-08-10 23:02:13 on dist-test-slave-6bbx"
server_key: "aa6c7444d3bfd8bb0d3b1ee2a59c666b"
server_key_iv: "014b30f6eac670cac6a48b2141220084"
server_key_version: "encryptionkey@0"
I20260810 23:02:13.739202 573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:13.752542 573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:13.755604 573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:13.755833 573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:13.756484 573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:02:13.757364 573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:02:13.757412 573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:13.757457 573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:02:13.757488 573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:13.766510 573 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:33691
I20260810 23:02:13.766582 754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:33691 every 8 connection(s)
I20260810 23:02:13.767508 573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:02:13.773927 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 573
I20260810 23:02:13.774034 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:02:13.774310 15584 external_mini_cluster.cc:1428] Setting key 80465e6ef995f291271134c88fb64c41
Aug 10 23:02:13 dist-test-slave-6bbx krb5kdc[418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402933, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:02:13.782219 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:13.769561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:49967 (local address 127.15.56.62:43641)
0810 23:02:13.769702 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:13.769707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:13.770661 (+ 954us) server_negotiation.cc:407] Connection header received
0810 23:02:13.772171 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:13.772175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:13.772235 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:13.772314 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:13.773563 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:13.774138 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:13.775440 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:13.775800 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:13.775899 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:13.778582 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:13.778586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:13.778588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:02:13.778614 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:13.780017 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:02:13.780563 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:02:13.780566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:02:13.780567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:02:13.780654 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:02:13.781013 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:02:13.781016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:02:13.781018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:02:13.781218 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:13.781318 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:13.781877 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 23:02:13.782025 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:02:13.783152 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:13.769999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43641 (local address 127.15.56.1:49967)
0810 23:02:13.770500 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:13.770548 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:13.771952 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:13.772509 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:13.772516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:13.772902 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:13.773401 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:13.773416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:13.774267 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:13.774270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:13.775319 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:13.775330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:13.775962 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:13.775966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:13.776147 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:13.776739 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:13.776759 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:13.778396 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:02:13.780140 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:02:13.780147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:02:13.780168 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:02:13.780458 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:02:13.780741 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:02:13.780744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:02:13.780747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:02:13.780917 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:02:13.781325 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:02:13.781332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:02:13.781620 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 23:02:13.781818 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 23:02:13.782224 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":138,"threads_started":1}
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:02:13.784667 755 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43641
I20260810 23:02:13.784941 755 heartbeater.cc:464] Registering TS with master...
I20260810 23:02:13.785527 755 heartbeater.cc:511] Master 127.15.56.62:43641 requested a full tablet report, sending...
I20260810 23:02:13.787459 485 ts_manager.cc:194] Registered new tserver with Master: 2da8daec96144034918df64101092513 (127.15.56.1:33691)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
I20260810 23:02:13.788770 485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:49967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:02:13.807801 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43641
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:02:13.924458 767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:13.924737 767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:13.924799 767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:13.928879 767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:13.928946 767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:13.929028 767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:02:13.934296 767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:13.935340 767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:13.936422 767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:02:13.946417 776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:13.946417 775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:13.946561 767 server_base.cc:1034] running on GCE node
W20260810 23:02:13.946735 779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:13.947178 767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:13.947969 767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:13.949167 767 hybrid_clock.cc:648] HybridClock initialized: now 1786402933949119 us; error 55 us; skew 500 ppm
Aug 10 23:02:13 dist-test-slave-6bbx krb5kdc[418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402933, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:02:13.952252 767 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:02:13.953222 767 webserver.cc:460] Webserver started at http://127.15.56.2:40941/ using document root <none> and password file <none>
I20260810 23:02:13.953805 767 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:13.953878 767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:13.954092 767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:13.956367 767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "ac52712549e044c49676d84e41e5ac46"
format_stamp: "Formatted at 2026-08-10 23:02:13 on dist-test-slave-6bbx"
server_key: "8be8e6ec0606a2b6e621bc46bf89c280"
server_key_iv: "026cd2d4887f5a6ad4e331a7f1806eda"
server_key_version: "encryptionkey@0"
I20260810 23:02:13.956934 767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "ac52712549e044c49676d84e41e5ac46"
format_stamp: "Formatted at 2026-08-10 23:02:13 on dist-test-slave-6bbx"
server_key: "8be8e6ec0606a2b6e621bc46bf89c280"
server_key_iv: "026cd2d4887f5a6ad4e331a7f1806eda"
server_key_version: "encryptionkey@0"
I20260810 23:02:13.960806 767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:02:13.963348 788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:13.964851 767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:02:13.964977 767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "ac52712549e044c49676d84e41e5ac46"
format_stamp: "Formatted at 2026-08-10 23:02:13 on dist-test-slave-6bbx"
server_key: "8be8e6ec0606a2b6e621bc46bf89c280"
server_key_iv: "026cd2d4887f5a6ad4e331a7f1806eda"
server_key_version: "encryptionkey@0"
I20260810 23:02:13.965092 767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:13.991437 767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:13.994763 767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:13.994973 767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:13.995652 767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:02:13.996600 767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:02:13.996654 767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:13.996711 767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:02:13.996738 767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:14.006042 767 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:34667
I20260810 23:02:14.006073 954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:34667 every 8 connection(s)
I20260810 23:02:14.007118 767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:02:14.015976 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 767
I20260810 23:02:14.016139 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:02:14.016479 15584 external_mini_cluster.cc:1428] Setting key a1c2ccc62c2c889ccc0b966c95a3e8aa
Aug 10 23:02:14 dist-test-slave-6bbx krb5kdc[418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402933, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:02:14.021757 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.009167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:37951 (local address 127.15.56.62:43641)
0810 23:02:14.009359 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:14.009364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:14.010362 (+ 998us) server_negotiation.cc:407] Connection header received
0810 23:02:14.011801 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:14.011804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:14.011856 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:14.011920 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:14.013161 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.013740 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.014982 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.015292 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.015417 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.018155 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:14.018158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:14.018160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:02:14.018184 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:14.019563 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:02:14.020202 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:02:14.020206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:02:14.020208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:02:14.020295 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:02:14.020639 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:02:14.020642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:02:14.020643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:02:14.020832 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:14.020919 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:14.021451 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 23:02:14.021577 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:02:14.022609 959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.009644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43641 (local address 127.15.56.2:37951)
0810 23:02:14.010152 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:14.010224 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:14.011542 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:14.012082 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:14.012090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:14.012502 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:14.013041 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.013056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.013849 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.013852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.014858 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.014871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.015451 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.015454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.015611 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.016275 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:14.016297 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:14.017985 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:02:14.019704 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:02:14.019712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:02:14.019730 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:02:14.020056 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:02:14.020382 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:02:14.020385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:02:14.020387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:02:14.020530 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:02:14.020936 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:02:14.020943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:02:14.021207 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 23:02:14.021373 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 23:02:14.021714 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":141,"threads_started":1}
I20260810 23:02:14.023763 956 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43641
I20260810 23:02:14.024058 956 heartbeater.cc:464] Registering TS with master...
I20260810 23:02:14.024691 956 heartbeater.cc:511] Master 127.15.56.62:43641 requested a full tablet report, sending...
I20260810 23:02:14.025902 485 ts_manager.cc:194] Registered new tserver with Master: ac52712549e044c49676d84e41e5ac46 (127.15.56.2:34667)
I20260810 23:02:14.026693 485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:37951
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:02:14.046480 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43641
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:02:14.162657 962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:02:14.162936 962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:02:14.162984 962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:02:14.167065 962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:02:14.167158 962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:02:14.167218 962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:02:14.173010 962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:02:14.174167 962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:02:14.175244 962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:02:14.185042 976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:14.185096 962 server_base.cc:1034] running on GCE node
W20260810 23:02:14.185288 980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:02:14.185101 975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:02:14.185755 962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:02:14.186444 962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:02:14.187605 962 hybrid_clock.cc:648] HybridClock initialized: now 1786402934187580 us; error 43 us; skew 500 ppm
Aug 10 23:02:14 dist-test-slave-6bbx krb5kdc[418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402934, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:02:14.190534 962 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:02:14.191473 962 webserver.cc:460] Webserver started at http://127.15.56.3:37561/ using document root <none> and password file <none>
I20260810 23:02:14.192083 962 fs_manager.cc:359] Metadata directory not provided
I20260810 23:02:14.192149 962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:02:14.192327 962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:02:14.194430 962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "fa2d9265478449c5b7053be4c4d875b2"
format_stamp: "Formatted at 2026-08-10 23:02:14 on dist-test-slave-6bbx"
server_key: "b3bc31688a35080da31528abcc94aa52"
server_key_iv: "aba1156289903f98dcc20d527221b3fd"
server_key_version: "encryptionkey@0"
I20260810 23:02:14.194973 962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "fa2d9265478449c5b7053be4c4d875b2"
format_stamp: "Formatted at 2026-08-10 23:02:14 on dist-test-slave-6bbx"
server_key: "b3bc31688a35080da31528abcc94aa52"
server_key_iv: "aba1156289903f98dcc20d527221b3fd"
server_key_version: "encryptionkey@0"
I20260810 23:02:14.198988 962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:02:14.201614 989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:14.202901 962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:02:14.203014 962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "fa2d9265478449c5b7053be4c4d875b2"
format_stamp: "Formatted at 2026-08-10 23:02:14 on dist-test-slave-6bbx"
server_key: "b3bc31688a35080da31528abcc94aa52"
server_key_iv: "aba1156289903f98dcc20d527221b3fd"
server_key_version: "encryptionkey@0"
I20260810 23:02:14.203132 962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:02:14.230154 962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:02:14.233510 962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:02:14.233704 962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:02:14.234355 962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:02:14.235216 962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:02:14.235266 962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:14.235309 962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:02:14.235337 962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:02:14.244475 962 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:36065
I20260810 23:02:14.244514 1177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:36065 every 8 connection(s)
I20260810 23:02:14.245420 962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:02:14.254204 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 962
I20260810 23:02:14.254354 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:02:14.254695 15584 external_mini_cluster.cc:1428] Setting key 99961b42a01f2227893f0281e6be8078
Aug 10 23:02:14 dist-test-slave-6bbx krb5kdc[418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402934, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:02:14.259552 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.247227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:37091 (local address 127.15.56.62:43641)
0810 23:02:14.247356 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:14.247359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:14.248316 (+ 957us) server_negotiation.cc:407] Connection header received
0810 23:02:14.249731 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:14.249735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:14.249793 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:14.249871 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:14.251125 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.251659 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.252837 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.253055 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.253148 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.255976 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:14.255979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:14.255981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:02:14.256007 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:14.257455 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:02:14.257935 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:02:14.257938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:02:14.257939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:02:14.258011 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:02:14.258376 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:02:14.258379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:02:14.258380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:02:14.258577 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:14.258678 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:14.259307 (+ 629us) server_negotiation.cc:299] Negotiation successful
0810 23:02:14.259423 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:02:14.260488 1181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.247639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43641 (local address 127.15.56.3:37091)
0810 23:02:14.248147 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:14.248193 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:14.249517 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:14.250065 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:14.250073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:14.250456 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:02:14.250976 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.250991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.251816 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.251819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.252714 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.252724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.253206 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.253210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.253376 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.253955 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:02:14.253977 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:02:14.255812 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:02:14.257565 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:02:14.257570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:02:14.257583 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:02:14.257852 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:02:14.258083 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:02:14.258087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:02:14.258089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:02:14.258258 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:02:14.258718 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:02:14.258724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:02:14.259008 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 23:02:14.259222 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 23:02:14.259525 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":128,"threads_started":1}
I20260810 23:02:14.261814 1178 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43641
I20260810 23:02:14.262069 1178 heartbeater.cc:464] Registering TS with master...
I20260810 23:02:14.262614 1178 heartbeater.cc:511] Master 127.15.56.62:43641 requested a full tablet report, sending...
I20260810 23:02:14.263744 485 ts_manager.cc:194] Registered new tserver with Master: fa2d9265478449c5b7053be4c4d875b2 (127.15.56.3:36065)
I20260810 23:02:14.264318 485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:37091
I20260810 23:02:14.268615 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:02:14.277477 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.270616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46900 (local address 127.15.56.62:43641)
0810 23:02:14.270731 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:14.270735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:14.270906 (+ 171us) server_negotiation.cc:407] Connection header received
0810 23:02:14.271020 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:14.271023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:14.271061 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:14.271118 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:02:14.271791 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.272300 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.273163 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.273386 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.273468 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.274723 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:02:14.274726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:02:14.274729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:02:14.274753 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:02:14.276091 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:02:14.276462 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:02:14.276464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:02:14.276466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:02:14.276538 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:02:14.276738 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:02:14.276740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:02:14.276741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:02:14.276990 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:02:14.277097 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:14.277248 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 23:02:14.277352 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 23:02:14.280296 485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:02:14.283344 485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:02:14.297792 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.292068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:34667 (local address 127.0.0.1:39682)
0810 23:02:14.292698 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:14.292754 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:14.292878 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:14.294017 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:14.294020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:14.294038 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:02:14.294130 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.294136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.295095 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.295099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.296280 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.296298 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.297214 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.297218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.297340 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.297383 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 23:02:14.297501 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 23:02:14.297637 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":102,"threads_started":1}
I20260810 23:02:14.297824 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.292388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39682 (local address 127.15.56.2:34667)
0810 23:02:14.293576 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:14.293587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:14.293609 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:02:14.293680 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:14.293685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:14.293884 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:14.294039 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:02:14.294222 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.294968 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.296402 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.297082 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.297193 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.297349 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:14.297543 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 23:02:14.297646 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1085,"thread_start_us":124,"threads_started":1}
I20260810 23:02:14.298161 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.292857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:33691 (local address 127.0.0.1:55508)
0810 23:02:14.293392 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:14.293404 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:14.293483 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:14.294122 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:14.294125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:14.294144 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:02:14.294223 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.294228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.295707 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.295710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.296809 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.296818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.297758 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.297762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.297840 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.297863 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:02:14.297919 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 23:02:14.297988 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":100,"threads_started":1}
I20260810 23:02:14.298177 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.292121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:36065 (local address 127.0.0.1:44662)
0810 23:02:14.293130 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:14.293142 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:14.293225 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:14.293814 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:14.293820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:14.293844 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:02:14.293979 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.293986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.295484 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.295487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.296736 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.296748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.297794 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.297798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.297887 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.297912 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:02:14.297965 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 23:02:14.298034 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260810 23:02:14.298666 1202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.292801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.15.56.3:36065)
0810 23:02:14.293280 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:14.293306 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:14.293331 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:02:14.293414 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:14.293419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:14.293645 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:14.293809 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:02:14.294123 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.295310 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.296871 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.297653 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.298254 (+ 601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.298414 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:14.298467 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 23:02:14.298551 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":150,"threads_started":1}
I20260810 23:02:14.298818 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.293291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.15.56.1:33691)
0810 23:02:14.293684 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:14.293689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:14.293706 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:02:14.293774 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:14.293779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:14.293970 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:14.294371 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:02:14.294505 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.295559 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.296926 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.297631 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.298414 (+ 783us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.298556 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:14.298601 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 23:02:14.298684 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":121,"threads_started":1}
I20260810 23:02:14.301631 670 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4177cc6bd94beea45ed0865bbb39fd (DEFAULT_TABLE table=test-table [id=434494831407458a9b38b0a790937b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:02:14.301492 1075 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4177cc6bd94beea45ed0865bbb39fd (DEFAULT_TABLE table=test-table [id=434494831407458a9b38b0a790937b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:02:14.301492 851 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4177cc6bd94beea45ed0865bbb39fd (DEFAULT_TABLE table=test-table [id=434494831407458a9b38b0a790937b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:02:14.302594 670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4177cc6bd94beea45ed0865bbb39fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:02:14.302629 1075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4177cc6bd94beea45ed0865bbb39fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:02:14.302621 851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4177cc6bd94beea45ed0865bbb39fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:02:14.309070 1206 tablet_bootstrap.cc:492] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513: Bootstrap starting.
I20260810 23:02:14.310550 1207 tablet_bootstrap.cc:492] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: Bootstrap starting.
I20260810 23:02:14.310680 1208 tablet_bootstrap.cc:492] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: Bootstrap starting.
I20260810 23:02:14.311537 1206 tablet_bootstrap.cc:654] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513: Neither blocks nor log segments found. Creating new log.
I20260810 23:02:14.312461 1206 log.cc:824] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513: Log is configured to *not* fsync() on all Append() calls
I20260810 23:02:14.312743 1207 tablet_bootstrap.cc:654] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: Neither blocks nor log segments found. Creating new log.
I20260810 23:02:14.313252 1208 tablet_bootstrap.cc:654] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: Neither blocks nor log segments found. Creating new log.
I20260810 23:02:14.313597 1207 log.cc:824] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: Log is configured to *not* fsync() on all Append() calls
I20260810 23:02:14.314059 1208 log.cc:824] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:02:14.314937 1206 tablet_bootstrap.cc:492] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513: No bootstrap required, opened a new log
I20260810 23:02:14.315145 1206 ts_tablet_manager.cc:1397] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:02:14.315867 1207 tablet_bootstrap.cc:492] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: No bootstrap required, opened a new log
I20260810 23:02:14.316053 1208 tablet_bootstrap.cc:492] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: No bootstrap required, opened a new log
I20260810 23:02:14.316061 1207 ts_tablet_manager.cc:1397] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:02:14.316246 1208 ts_tablet_manager.cc:1397] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:02:14.319178 1208 raft_consensus.cc:348] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac52712549e044c49676d84e41e5ac46" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34667 } } peers { permanent_uuid: "fa2d9265478449c5b7053be4c4d875b2" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36065 } } peers { permanent_uuid: "2da8daec96144034918df64101092513" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33691 } }
I20260810 23:02:14.319407 1208 raft_consensus.cc:374] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:02:14.319479 1208 raft_consensus.cc:729] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa2d9265478449c5b7053be4c4d875b2, State: Initialized, Role: FOLLOWER
I20260810 23:02:14.319783 1207 raft_consensus.cc:348] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac52712549e044c49676d84e41e5ac46" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34667 } } peers { permanent_uuid: "fa2d9265478449c5b7053be4c4d875b2" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36065 } } peers { permanent_uuid: "2da8daec96144034918df64101092513" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33691 } }
I20260810 23:02:14.319783 1206 raft_consensus.cc:348] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac52712549e044c49676d84e41e5ac46" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34667 } } peers { permanent_uuid: "fa2d9265478449c5b7053be4c4d875b2" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36065 } } peers { permanent_uuid: "2da8daec96144034918df64101092513" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33691 } }
I20260810 23:02:14.319968 1208 consensus_queue.cc:260] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac52712549e044c49676d84e41e5ac46" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34667 } } peers { permanent_uuid: "fa2d9265478449c5b7053be4c4d875b2" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36065 } } peers { permanent_uuid: "2da8daec96144034918df64101092513" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33691 } }
I20260810 23:02:14.320050 1206 raft_consensus.cc:374] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:02:14.320050 1207 raft_consensus.cc:374] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:02:14.320101 1207 raft_consensus.cc:729] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac52712549e044c49676d84e41e5ac46, State: Initialized, Role: FOLLOWER
I20260810 23:02:14.320111 1206 raft_consensus.cc:729] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da8daec96144034918df64101092513, State: Initialized, Role: FOLLOWER
I20260810 23:02:14.320585 1207 consensus_queue.cc:260] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac52712549e044c49676d84e41e5ac46" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34667 } } peers { permanent_uuid: "fa2d9265478449c5b7053be4c4d875b2" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36065 } } peers { permanent_uuid: "2da8daec96144034918df64101092513" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33691 } }
I20260810 23:02:14.320585 1206 consensus_queue.cc:260] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac52712549e044c49676d84e41e5ac46" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34667 } } peers { permanent_uuid: "fa2d9265478449c5b7053be4c4d875b2" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36065 } } peers { permanent_uuid: "2da8daec96144034918df64101092513" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33691 } }
I20260810 23:02:14.321038 1208 ts_tablet_manager.cc:1428] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:02:14.321470 1178 heartbeater.cc:503] Master 127.15.56.62:43641 was elected leader, sending a full tablet report...
I20260810 23:02:14.321576 1206 ts_tablet_manager.cc:1428] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:02:14.321576 1207 ts_tablet_manager.cc:1428] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:02:14.321940 755 heartbeater.cc:503] Master 127.15.56.62:43641 was elected leader, sending a full tablet report...
I20260810 23:02:14.322918 956 heartbeater.cc:503] Master 127.15.56.62:43641 was elected leader, sending a full tablet report...
I20260810 23:02:14.475302 1218 raft_consensus.cc:482] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:02:14.475533 1218 raft_consensus.cc:504] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac52712549e044c49676d84e41e5ac46" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34667 } } peers { permanent_uuid: "fa2d9265478449c5b7053be4c4d875b2" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36065 } } peers { permanent_uuid: "2da8daec96144034918df64101092513" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33691 } }
I20260810 23:02:14.476814 1218 leader_election.cc:288] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa2d9265478449c5b7053be4c4d875b2 (127.15.56.3:36065), 2da8daec96144034918df64101092513 (127.15.56.1:33691)
I20260810 23:02:14.481542 1202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.477202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:49543 (local address 127.15.56.3:36065)
0810 23:02:14.477467 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:14.477471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:14.477625 (+ 154us) server_negotiation.cc:407] Connection header received
0810 23:02:14.477800 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:14.477802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:14.477843 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:14.477905 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:02:14.478163 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.478744 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.480191 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.480796 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.480913 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.480982 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:14.481341 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 23:02:14.481418 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260810 23:02:14.481544 1203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.477213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:37331 (local address 127.15.56.1:33691)
0810 23:02:14.477374 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:14.477378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:14.477649 (+ 271us) server_negotiation.cc:407] Connection header received
0810 23:02:14.477810 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:14.477813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:14.477867 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:14.477948 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:02:14.478214 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.478807 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.480188 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.480835 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.480929 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.480982 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:14.481334 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 23:02:14.481423 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 23:02:14.481541 959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.477130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:36065 (local address 127.15.56.2:49543)
0810 23:02:14.477541 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:14.477553 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:14.477697 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:14.477924 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:14.477927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:14.477942 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:02:14.478058 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.478064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.478846 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.478850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.480039 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.480050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.480974 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.480977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.481139 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.481191 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 23:02:14.481272 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:02:14.481362 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":202,"spinlock_wait_cycles":46848,"thread_start_us":109,"threads_started":1}
I20260810 23:02:14.481549 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.477130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:33691 (local address 127.15.56.2:37331)
0810 23:02:14.477541 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:14.477554 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:02:14.477700 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:02:14.477966 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:02:14.477969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:02:14.477985 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:02:14.478107 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.478112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.478918 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.478922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.480038 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:02:14.480050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.480973 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.480977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:02:14.481138 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.481187 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 23:02:14.481272 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:02:14.481362 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":51,"spinlock_wait_cycles":5248}
I20260810 23:02:14.482345 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4177cc6bd94beea45ed0865bbb39fd" candidate_uuid: "ac52712549e044c49676d84e41e5ac46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa2d9265478449c5b7053be4c4d875b2" is_pre_election: true
I20260810 23:02:14.482570 701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4177cc6bd94beea45ed0865bbb39fd" candidate_uuid: "ac52712549e044c49676d84e41e5ac46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da8daec96144034918df64101092513" is_pre_election: true
I20260810 23:02:14.482683 1110 raft_consensus.cc:2454] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac52712549e044c49676d84e41e5ac46 in term 0.
I20260810 23:02:14.482851 701 raft_consensus.cc:2454] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac52712549e044c49676d84e41e5ac46 in term 0.
I20260810 23:02:14.483174 790 leader_election.cc:302] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac52712549e044c49676d84e41e5ac46, fa2d9265478449c5b7053be4c4d875b2; no voters:
I20260810 23:02:14.483467 1218 raft_consensus.cc:2781] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:02:14.483551 1218 raft_consensus.cc:482] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:02:14.483582 1218 raft_consensus.cc:3037] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:02:14.484661 1218 raft_consensus.cc:504] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac52712549e044c49676d84e41e5ac46" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34667 } } peers { permanent_uuid: "fa2d9265478449c5b7053be4c4d875b2" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36065 } } peers { permanent_uuid: "2da8daec96144034918df64101092513" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33691 } }
I20260810 23:02:14.485008 1218 leader_election.cc:288] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [CANDIDATE]: Term 1 election: Requested vote from peers fa2d9265478449c5b7053be4c4d875b2 (127.15.56.3:36065), 2da8daec96144034918df64101092513 (127.15.56.1:33691)
I20260810 23:02:14.485368 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4177cc6bd94beea45ed0865bbb39fd" candidate_uuid: "ac52712549e044c49676d84e41e5ac46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa2d9265478449c5b7053be4c4d875b2"
I20260810 23:02:14.485370 701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4177cc6bd94beea45ed0865bbb39fd" candidate_uuid: "ac52712549e044c49676d84e41e5ac46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da8daec96144034918df64101092513"
I20260810 23:02:14.485477 1110 raft_consensus.cc:3037] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:02:14.485486 701 raft_consensus.cc:3037] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:02:14.486557 701 raft_consensus.cc:2454] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac52712549e044c49676d84e41e5ac46 in term 1.
I20260810 23:02:14.486605 1110 raft_consensus.cc:2454] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac52712549e044c49676d84e41e5ac46 in term 1.
I20260810 23:02:14.486830 792 leader_election.cc:302] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da8daec96144034918df64101092513, ac52712549e044c49676d84e41e5ac46; no voters:
I20260810 23:02:14.487011 1218 raft_consensus.cc:2781] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:02:14.487236 1218 raft_consensus.cc:686] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 1 LEADER]: Becoming Leader. State: Replica: ac52712549e044c49676d84e41e5ac46, State: Running, Role: LEADER
I20260810 23:02:14.487576 1218 consensus_queue.cc:237] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac52712549e044c49676d84e41e5ac46" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34667 } } peers { permanent_uuid: "fa2d9265478449c5b7053be4c4d875b2" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36065 } } peers { permanent_uuid: "2da8daec96144034918df64101092513" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33691 } }
I20260810 23:02:14.490907 484 catalog_manager.cc:5495] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac52712549e044c49676d84e41e5ac46 (127.15.56.2). New cstate: current_term: 1 leader_uuid: "ac52712549e044c49676d84e41e5ac46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac52712549e044c49676d84e41e5ac46" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa2d9265478449c5b7053be4c4d875b2" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2da8daec96144034918df64101092513" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33691 } health_report { overall_health: UNKNOWN } } }
W20260810 23:02:14.496376 1179 tablet.cc:2369] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:02:14.508392 957 tablet.cc:2369] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:02:14.519111 757 tablet.cc:2369] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:02:14.533077 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.529922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39690 (local address 127.15.56.2:34667)
0810 23:02:14.530058 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:02:14.530061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:02:14.530075 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:02:14.530155 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:02:14.530158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:02:14.530219 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:02:14.530308 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:02:14.530500 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.531002 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.531802 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:02:14.532021 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:02:14.532091 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:02:14.532317 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:02:14.532745 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:02:14.532837 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:02:14.532873 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 23:02:14.532926 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:02:14.539091 701 raft_consensus.cc:1261] T 1b4177cc6bd94beea45ed0865bbb39fd P 2da8daec96144034918df64101092513 [term 1 FOLLOWER]: Refusing update from remote peer ac52712549e044c49676d84e41e5ac46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:02:14.539093 1110 raft_consensus.cc:1261] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2 [term 1 FOLLOWER]: Refusing update from remote peer ac52712549e044c49676d84e41e5ac46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:02:14.539821 1222 consensus_queue.cc:1035] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa2d9265478449c5b7053be4c4d875b2" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 36065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:02:14.540021 1218 consensus_queue.cc:1035] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da8daec96144034918df64101092513" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:02:14.549026 1228 mvcc.cc:205] Tried to move back new op lower bound from 7317106419864596480 to 7317106419663831040. Current Snapshot: MvccSnapshot[applied={T|T < 7317106419864596480}]
I20260810 23:02:14.550958 1232 mvcc.cc:205] Tried to move back new op lower bound from 7317106419864596480 to 7317106419663831040. Current Snapshot: MvccSnapshot[applied={T|T < 7317106419864596480}]
I20260810 23:02:14.557085 484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:02:14.557302 484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:02:14.559945 484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3666b81e221474090795f578ae80862: Sending DeleteTablet for 3 replicas of tablet 1b4177cc6bd94beea45ed0865bbb39fd
I20260810 23:02:14.560879 1075 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b4177cc6bd94beea45ed0865bbb39fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:02:14 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:44662
I20260810 23:02:14.560873 670 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b4177cc6bd94beea45ed0865bbb39fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:02:14 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:55508
I20260810 23:02:14.560976 851 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b4177cc6bd94beea45ed0865bbb39fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:02:14 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:39682
I20260810 23:02:14.561478 1242 tablet_replica.cc:330] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: stopping tablet replica
I20260810 23:02:14.561661 1240 tablet_replica.cc:330] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: stopping tablet replica
I20260810 23:02:14.561883 1242 raft_consensus.cc:2229] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:02:14.562027 1240 raft_consensus.cc:2229] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:02:14.562081 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 573
I20260810 23:02:14.562446 1240 raft_consensus.cc:2258] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:02:14.565599 1242 raft_consensus.cc:2258] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:02:14.566380 1240 ts_tablet_manager.cc:1905] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:02:14.567090 1242 ts_tablet_manager.cc:1905] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:02:14.569623 1240 ts_tablet_manager.cc:1918] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:02:14.569923 1240 log.cc:1196] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/1b4177cc6bd94beea45ed0865bbb39fd
I20260810 23:02:14.570046 1242 ts_tablet_manager.cc:1918] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:02:14.570178 1242 log.cc:1196] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/1b4177cc6bd94beea45ed0865bbb39fd
I20260810 23:02:14.570310 1240 ts_tablet_manager.cc:1939] T 1b4177cc6bd94beea45ed0865bbb39fd P fa2d9265478449c5b7053be4c4d875b2: Deleting consensus metadata
I20260810 23:02:14.570513 1242 ts_tablet_manager.cc:1939] T 1b4177cc6bd94beea45ed0865bbb39fd P ac52712549e044c49676d84e41e5ac46: Deleting consensus metadata
I20260810 23:02:14.571777 466 catalog_manager.cc:4841] TS ac52712549e044c49676d84e41e5ac46 (127.15.56.2:34667): tablet 1b4177cc6bd94beea45ed0865bbb39fd (table test-table [id=434494831407458a9b38b0a790937b27]) successfully deleted
I20260810 23:02:14.571805 464 catalog_manager.cc:4841] TS fa2d9265478449c5b7053be4c4d875b2 (127.15.56.3:36065): tablet 1b4177cc6bd94beea45ed0865bbb39fd (table test-table [id=434494831407458a9b38b0a790937b27]) successfully deleted
W20260810 23:02:14.579156 466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:33691 (error 108)
I20260810 23:02:14.579355 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 767
I20260810 23:02:14.584478 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:02:14.584175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:33691 (local address 127.0.0.1:55510)
0810 23:02:14.584314 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:02:14.584387 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:33691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:02:14.584796 466 catalog_manager.cc:4584] TS 2da8daec96144034918df64101092513 (127.15.56.1:33691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b4177cc6bd94beea45ed0865bbb39fd: Network error: Client connection negotiation failed: client connection to 127.15.56.1:33691: connect: Connection refused (error 111)
I20260810 23:02:14.590754 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 962
I20260810 23:02:14.601483 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 438
2026-08-10T23:02:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2618 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:02:14 dist-test-slave-6bbx krb5kdc[1247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:02:14 dist-test-slave-6bbx krb5kdc[1247](info): set up 2 sockets
Aug 10 23:02:14 dist-test-slave-6bbx krb5kdc[1247](info): commencing operation
krb5kdc: starting...
W20260810 23:02:15.758554 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:02:15 dist-test-slave-6bbx krb5kdc[1247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:02:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/rangeradmin_127.15.56.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/rangeradmin_127.15.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/rangerlookup_127.15.56.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/rangerlookup_127.15.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/HTTP_127.15.56.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/HTTP_127.15.56.20@KRBTEST.COM.keytab.
I20260810 23:02:15.947141 15584 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWzUZ3m/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/postgres -l logfile start
2026-08-10 23:02:18.676 UTC [1287] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:02:18.677 UTC [1287] LOG: listening on IPv4 address "127.15.56.20", port 46571
2026-08-10 23:02:18.680 UTC [1287] LOG: listening on Unix socket "/tmp/.s.PGSQL.46571"
2026-08-10 23:02:18.701 UTC [1290] LOG: database system was shut down at 2026-08-10 23:02:17 UTC
2026-08-10 23:02:18.704 UTC [1287] LOG: database system is ready to accept connections
I20260810 23:02:19.791882 15584 mini_postgres.cc:96] Postgres bound to 46571
2026-08-10 23:02:19.800 UTC [1306] FATAL: database "slave" does not exist
127.15.56.20:46571 - accepting connections
I20260810 23:02:19.801224 15584 mini_ranger.cc:161] Starting Ranger...
I20260810 23:02:19.817754 15584 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:02:20.390123 15584 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:02:20.390275 15584 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-admin
2026-08-10 23:02:20,811 [I] DB FLAVOR :POSTGRES
2026-08-10 23:02:20,812 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:02:20,812 [I] Checking connection..
2026-08-10 23:02:20,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:02:21,113 [I] Checking connection passed.
2026-08-10 23:02:21,113 [I] --------- Verifying version history table ---------
2026-08-10 23:02:21,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:02:21,377 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:02:21,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:02:21,639 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:02:21,639 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:02:21,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:02:21.892 UTC [1397] WARNING: there is no transaction in progress
2026-08-10 23:02:21,902 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:02:21,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:02:22,160 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:02:22,160 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:02:22,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:02:22,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:02:22,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6bbx.c.gcp-upstream.internal','N') ;"
2026-08-10 23:02:22,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:02:23,215 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:02:23,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:02:23,471 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:02:23,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:02:23,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:02:23,991 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:02:23,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:02:25.019 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.036 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.053 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.070 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.091 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.112 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.169 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.176 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.193 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.204 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.215 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.231 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.242 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.249 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.255 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25.262 UTC [1547] WARNING: there is no transaction in progress
2026-08-10 23:02:25,957 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:02:25,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:02:26,219 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:02:26,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:02:26,479 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:02:26,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:02:26,729 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:02:26,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:02:26,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:02:27,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6bbx.c.gcp-upstream.internal';"
2026-08-10 23:02:27,484 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:02:27,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:02:27,756 [I] DB_PATCHES have already been applied
I20260810 23:02:27.762280 15584 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWzUZ3m/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:02:27.762354 15584 mini_ranger.cc:193] Using host: 127.15.56.20
I20260810 23:02:27.766855 15584 mini_ranger.cc:227] Ranger admin URL: http://127.15.56.20:53233
Aug 10, 2026 11:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:02:30.310506 15584 mini_ranger.cc:160] Time spent starting Ranger: real 10.509s user 0.000s sys 0.014s
Aug 10, 2026 11:02:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:02:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 11:02:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:02:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:02:51 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:02:52.628840 15584 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/rangerkms_127.15.56.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/rangerkms_127.15.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.56.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.56.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.56.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/HTTP_127.15.56.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.56.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/HTTP_127.15.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:02:52 dist-test-slave-6bbx krb5kdc[1247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402972, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:02:52.721374 15584 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:02:52.732302 15584 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:02:53.610486 15584 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:02:53.610574 15584 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-kms
I20260810 23:02:53.610658 15584 mini_ranger_kms.cc:227] Using postgres at 127.15.56.20:46571
2026-08-10 23:02:54,107 [I] DB FLAVOR :POSTGRES
2026-08-10 23:02:54,107 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:02:54,107 [I] Checking connection
2026-08-10 23:02:54,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:02:54,395 [I] connection success
2026-08-10 23:02:54,395 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:02:54,396 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:02:54,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:02:54,660 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:02:54,661 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:02:54,661 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:02:54,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:46571/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:02:54,959 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:02:55.490183 15584 mini_ranger_kms.cc:310] Created kms service
I20260810 23:02:55.715868 15584 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:02:55.763372 15584 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:02:55 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402972, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:02:56.210100 15584 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:02:56.210469 15584 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:02:56.210496 15584 mini_ranger_kms.cc:240] Using host: 127.15.56.20
I20260810 23:02:56.216449 15584 mini_ranger_kms.cc:276] Ranger KMS PID: 1825
I20260810 23:02:56.216607 15584 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.56.20:56583
Aug 10, 2026 11:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 23:02:58 dist-test-slave-6bbx krb5kdc[1247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786402978, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.56.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:02:58.761430 15584 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.040s user 0.018s sys 0.231s
I20260810 23:02:58.761592 15584 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:02:58.761744 15584 mini_ranger_kms.cc:401] 127.15.56.20:56583/kms/v1/keys
Aug 10 23:02:58 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402978, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.56.20@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
Aug 10 23:02:59 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402978, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.56.20@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
Aug 10 23:03:01 dist-test-slave-6bbx krb5kdc[1247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:03:01.509155 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44421
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:44421
--ranger_config_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:56583/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:01.628912 1900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:01.629232 1900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:01.629292 1900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:01.633466 1900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:01.633559 1900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:01.633587 1900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:01.633608 1900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:01.633627 1900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:01.639945 1900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.56.20:56583/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:44421
--ranger_config_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:01.641150 1900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:01.642284 1900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:01.651758 1909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:01.651758 1910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:01.651927 1900 server_base.cc:1034] running on GCE node
W20260810 23:03:01.651777 1912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:01.652653 1900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:01.653805 1900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:01.654996 1900 hybrid_clock.cc:648] HybridClock initialized: now 1786402981654975 us; error 39 us; skew 500 ppm
Aug 10 23:03:01 dist-test-slave-6bbx krb5kdc[1247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402981, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:01.657984 1900 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:03:01.658953 1900 webserver.cc:460] Webserver started at http://127.15.56.62:35793/ using document root <none> and password file <none>
I20260810 23:03:01.659499 1900 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:01.659555 1900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:01.659806 1900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:03:01 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402981, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:01.722666 1900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "19faab31330d4664b93d9e79a3a3a3ca"
format_stamp: "Formatted at 2026-08-10 23:03:01 on dist-test-slave-6bbx"
server_key: "831358d02c09258c105ddcb86c51a1af"
server_key_iv: "60a5df6933a94168a424e5c591dd2bb9"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:01.723302 1900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "19faab31330d4664b93d9e79a3a3a3ca"
format_stamp: "Formatted at 2026-08-10 23:03:01 on dist-test-slave-6bbx"
server_key: "831358d02c09258c105ddcb86c51a1af"
server_key_iv: "60a5df6933a94168a424e5c591dd2bb9"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:01.727368 1900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 23:03:01.744028 1921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:01.746057 1900 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260810 23:03:01.746250 1900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "19faab31330d4664b93d9e79a3a3a3ca"
format_stamp: "Formatted at 2026-08-10 23:03:01 on dist-test-slave-6bbx"
server_key: "831358d02c09258c105ddcb86c51a1af"
server_key_iv: "60a5df6933a94168a424e5c591dd2bb9"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:01.746421 1900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:01.772027 1900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:01.775707 1900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:01.775904 1900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:01.783840 1900 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:44421
I20260810 23:03:01.783856 2000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:44421 every 8 connection(s)
I20260810 23:03:01.784929 1900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:01.788043 2001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:01.788975 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 1900
I20260810 23:03:01.789100 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:03:01.789434 15584 external_mini_cluster.cc:1428] Setting key a93972fa06230fa63a77f692467b8b85
I20260810 23:03:01.795011 2001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca: Bootstrap starting.
I20260810 23:03:01.797623 2001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:01.798501 2001 log.cc:824] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:03:01 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:03:01.801157 2001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca: No bootstrap required, opened a new log
I20260810 23:03:01.805068 2001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19faab31330d4664b93d9e79a3a3a3ca" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44421 } }
I20260810 23:03:01.805351 2001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:01.805414 2001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19faab31330d4664b93d9e79a3a3a3ca, State: Initialized, Role: FOLLOWER
I20260810 23:03:01.805708 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:01.791073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.15.56.62:44421)
0810 23:03:01.791580 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:01.791597 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:01.791636 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:03:01.792431 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:01.792451 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:01.792820 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:01.793198 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:01.794016 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:01.795031 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:01.796040 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:01.796542 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:01.796719 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:01.799466 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:01.799474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:01.799499 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:01.799540 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:01.802328 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:01.802927 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:01.802932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:01.802937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:01.803057 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:01.803384 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:01.803387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:01.803388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:01.803962 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:01.804136 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:01.804452 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 23:03:01.804761 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":159,"threads_started":1}
I20260810 23:03:01.805945 2001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19faab31330d4664b93d9e79a3a3a3ca" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44421 } }
I20260810 23:03:01.806116 2001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:01.806187 2001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:01.806298 2001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:01.807420 2001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19faab31330d4664b93d9e79a3a3a3ca" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44421 } }
I20260810 23:03:01.807819 2001 leader_election.cc:302] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19faab31330d4664b93d9e79a3a3a3ca; no voters:
I20260810 23:03:01.808203 2001 leader_election.cc:288] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:01.808313 2007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:01.808578 2007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [term 1 LEADER]: Becoming Leader. State: Replica: 19faab31330d4664b93d9e79a3a3a3ca, State: Running, Role: LEADER
I20260810 23:03:01.808898 2007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19faab31330d4664b93d9e79a3a3a3ca" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44421 } }
I20260810 23:03:01.809605 2001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:01.810799 2009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19faab31330d4664b93d9e79a3a3a3ca. Latest consensus state: current_term: 1 leader_uuid: "19faab31330d4664b93d9e79a3a3a3ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19faab31330d4664b93d9e79a3a3a3ca" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44421 } } }
I20260810 23:03:01.810799 2010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19faab31330d4664b93d9e79a3a3a3ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19faab31330d4664b93d9e79a3a3a3ca" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44421 } } }
I20260810 23:03:01.810964 2010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:01.810964 2009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:01.811378 2021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:01.814833 2021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:01.815505 2001 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:03:01.825233 2021 catalog_manager.cc:1347] Generated new cluster ID: 7e8d802f59e14dc5a9c820e4877e543c
I20260810 23:03:01.825340 2021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:01.850905 2021 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:01.852027 2021 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:01.857807 2021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca: Generated new TSK 0
I20260810 23:03:01.858520 2021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:03:02.563771 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:56583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44421
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:02.718961 2070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:02.719236 2070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:02.719280 2070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:02.724592 2070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:02.724690 2070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:02.724781 2070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:03:02.732573 2070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.56.20:56583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:02.733851 2070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:02.734982 2070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:02.748178 2086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:02.748296 2084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:02.748621 2083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:02.748939 2070 server_base.cc:1034] running on GCE node
I20260810 23:03:02.749492 2070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:02.750365 2070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:02.751564 2070 hybrid_clock.cc:648] HybridClock initialized: now 1786402982751517 us; error 60 us; skew 500 ppm
Aug 10 23:03:02 dist-test-slave-6bbx krb5kdc[1247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402982, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:02.755079 2070 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:03:02.756227 2070 webserver.cc:460] Webserver started at http://127.15.56.1:36131/ using document root <none> and password file <none>
I20260810 23:03:02.756852 2070 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:02.756922 2070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:02.757141 2070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:03:02 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402982, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:02.772917 2070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "1d00660c68ba4d7381f2f1000f865294"
format_stamp: "Formatted at 2026-08-10 23:03:02 on dist-test-slave-6bbx"
server_key: "3150149644cfc51e2504ba45bd2e170c"
server_key_iv: "3ded8d7a09861c327592d853c41b0ff9"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:02.778534 2070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "1d00660c68ba4d7381f2f1000f865294"
format_stamp: "Formatted at 2026-08-10 23:03:02 on dist-test-slave-6bbx"
server_key: "3150149644cfc51e2504ba45bd2e170c"
server_key_iv: "3ded8d7a09861c327592d853c41b0ff9"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:02.785130 2070 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 23:03:02.807174 2099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:02.809197 2070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 23:03:02.809404 2070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "1d00660c68ba4d7381f2f1000f865294"
format_stamp: "Formatted at 2026-08-10 23:03:02 on dist-test-slave-6bbx"
server_key: "3150149644cfc51e2504ba45bd2e170c"
server_key_iv: "3ded8d7a09861c327592d853c41b0ff9"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:02.809574 2070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:02.828895 2070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:02.833159 2070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:02.833477 2070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:02.834368 2070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:02.835623 2070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:02.835796 2070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:02.835945 2070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:02.836038 2070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:02.847519 2070 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:45623
I20260810 23:03:02.847569 2271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:45623 every 8 connection(s)
I20260810 23:03:02.849010 2070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:02.856459 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 2070
I20260810 23:03:02.856600 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:02.856925 15584 external_mini_cluster.cc:1428] Setting key 1b7a3ebc6ee5ef340f2e906f97043d26
Aug 10 23:03:02 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402982, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:03:02.869756 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:02.851360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:42599 (local address 127.15.56.62:44421)
0810 23:03:02.851734 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:02.851745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:02.852884 (+ 1139us) server_negotiation.cc:407] Connection header received
0810 23:03:02.854415 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:02.854419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:02.854486 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:02.854561 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:02.856472 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:02.857347 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:02.860681 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:02.861044 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:02.861132 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:02.864631 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:02.864636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:02.864639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:02.864675 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:02.866708 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:02.867455 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:02.867459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:02.867461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:02.867576 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:02.868115 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:02.868119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:02.868120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:02.868410 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:02.868507 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:02.869385 (+ 878us) server_negotiation.cc:299] Negotiation successful
0810 23:03:02.869552 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:03:02.872035 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:02.851984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44421 (local address 127.15.56.1:42599)
0810 23:03:02.852691 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:02.852759 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:02.854113 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:02.855057 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:02.855068 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:02.855466 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:02.856281 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:02.856304 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:02.858875 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:02.858878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:02.860505 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:02.860520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:02.861414 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:02.861419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:02.861651 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:02.862439 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:02.862462 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:02.864368 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:02.866852 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:02.866860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:02.866884 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:02.867295 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:02.867735 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:02.867738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:02.867740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:02.867983 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:02.868589 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:02.868596 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:02.869036 (+ 440us) client_negotiation.cc:769] Sending connection context
0810 23:03:02.870319 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0810 23:03:02.870761 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":169,"threads_started":1}
I20260810 23:03:02.874092 2272 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44421
I20260810 23:03:02.874457 2272 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
I20260810 23:03:02.875288 2272 heartbeater.cc:511] Master 127.15.56.62:44421 requested a full tablet report, sending...
I20260810 23:03:02.877470 1949 ts_manager.cc:194] Registered new tserver with Master: 1d00660c68ba4d7381f2f1000f865294 (127.15.56.1:45623)
I20260810 23:03:02.879201 1949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:42599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:03:02.900460 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:56583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44421
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:03:02 dist-test-slave-6bbx krb5kdc[1247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786402982, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 23:03:03.040107 2279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:03.040422 2279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:03.040484 2279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:03.044819 2279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:03.044898 2279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:03.044982 2279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:03:03.050782 2279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.56.20:56583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:03.052018 2279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:03.053196 2279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:03.065306 2292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:03.065452 2289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:03.065306 2290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:03.066632 2279 server_base.cc:1034] running on GCE node
I20260810 23:03:03.067435 2279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:03.068327 2279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:03.069519 2279 hybrid_clock.cc:648] HybridClock initialized: now 1786402983069457 us; error 86 us; skew 500 ppm
Aug 10 23:03:03 dist-test-slave-6bbx krb5kdc[1247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402983, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:03.072760 2279 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:03:03.073968 2279 webserver.cc:460] Webserver started at http://127.15.56.2:42791/ using document root <none> and password file <none>
I20260810 23:03:03.074743 2279 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:03.074875 2279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:03.075168 2279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:03:03 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402983, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:03.105168 2279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "74daeff6ac984d30a4033e49dcd06c6b"
format_stamp: "Formatted at 2026-08-10 23:03:03 on dist-test-slave-6bbx"
server_key: "6582ff4c1ca73370523c88bd4c2bc317"
server_key_iv: "849509c2a81313c610189e924225049d"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:03.105834 2279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "74daeff6ac984d30a4033e49dcd06c6b"
format_stamp: "Formatted at 2026-08-10 23:03:03 on dist-test-slave-6bbx"
server_key: "6582ff4c1ca73370523c88bd4c2bc317"
server_key_iv: "849509c2a81313c610189e924225049d"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:03.110746 2279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 23:03:03.130597 2304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:03.132623 2279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 23:03:03.132793 2279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "74daeff6ac984d30a4033e49dcd06c6b"
format_stamp: "Formatted at 2026-08-10 23:03:03 on dist-test-slave-6bbx"
server_key: "6582ff4c1ca73370523c88bd4c2bc317"
server_key_iv: "849509c2a81313c610189e924225049d"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:03.132961 2279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:03.162385 2279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:03.166535 2279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:03.166756 2279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:03.167505 2279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:03.168694 2279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:03.168762 2279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:03.168821 2279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:03.168853 2279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:03.182916 2279 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:44517
I20260810 23:03:03.184594 2279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:03.185392 2473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:44517 every 8 connection(s)
I20260810 23:03:03.193753 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 2279
I20260810 23:03:03.193909 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:03.194209 15584 external_mini_cluster.cc:1428] Setting key 4fa8d566368d195a7816a2976601e93d
Aug 10 23:03:03 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402983, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:03:03.213318 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:03.189318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:43787 (local address 127.15.56.62:44421)
0810 23:03:03.189458 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:03.189463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:03.198470 (+ 9007us) server_negotiation.cc:407] Connection header received
0810 23:03:03.199900 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:03.199905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:03.199968 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:03.200022 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:03.201947 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.202749 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.204645 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.204998 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.205080 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:03.208454 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:03.208459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:03.208462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:03.208499 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:03.210332 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:03.211131 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:03.211135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:03.211137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:03.211249 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:03.211790 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:03.211794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:03.211796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:03.212049 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:03.212147 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:03.212950 (+ 803us) server_negotiation.cc:299] Negotiation successful
0810 23:03:03.213104 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:03:03.214926 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:03.187991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44421 (local address 127.15.56.2:43787)
0810 23:03:03.198190 (+ 10199us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:03.198285 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:03.199550 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:03.200538 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:03.200550 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:03.200974 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:03.201715 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:03.201738 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.202937 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.202940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:03.204407 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:03.204444 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.205317 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.205321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:03.205534 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:03.206295 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:03.206318 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:03.208198 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:03.210482 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:03.210491 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:03.210516 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:03.211003 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:03.211396 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:03.211400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:03.211402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:03.211653 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:03.212166 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:03.212175 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:03.212612 (+ 437us) client_negotiation.cc:769] Sending connection context
0810 23:03:03.212875 (+ 263us) client_negotiation.cc:240] Negotiation successful
0810 23:03:03.213266 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9870,"thread_start_us":204,"threads_started":1}
I20260810 23:03:03.216588 2474 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44421
I20260810 23:03:03.216902 2474 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:03.217584 2474 heartbeater.cc:511] Master 127.15.56.62:44421 requested a full tablet report, sending...
I20260810 23:03:03.219007 1949 ts_manager.cc:194] Registered new tserver with Master: 74daeff6ac984d30a4033e49dcd06c6b (127.15.56.2:44517)
I20260810 23:03:03.219785 1949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:43787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:03:03.265581 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:56583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44421
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:03:03 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402982, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
W20260810 23:03:03.445381 2484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:03.445855 2484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:03.445977 2484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:03.452641 2484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:03.452781 2484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:03.452909 2484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:03:03.461055 2484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.56.20:56583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:03.462725 2484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:03.464214 2484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:03:03.479243 2484 server_base.cc:1034] running on GCE node
W20260810 23:03:03.479537 2498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:03.479856 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:03.485899 2497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:03.486665 2484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:03.487622 2484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:03.489023 2484 hybrid_clock.cc:648] HybridClock initialized: now 1786402983488945 us; error 98 us; skew 500 ppm
Aug 10 23:03:03 dist-test-slave-6bbx krb5kdc[1247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402983, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:03.492841 2484 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:03:03.494118 2484 webserver.cc:460] Webserver started at http://127.15.56.3:35899/ using document root <none> and password file <none>
I20260810 23:03:03.494949 2484 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:03.495103 2484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:03.495437 2484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:03:03 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402983, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:03.541878 2484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd"
format_stamp: "Formatted at 2026-08-10 23:03:03 on dist-test-slave-6bbx"
server_key: "1811299e44efc02ca959f5ead5cd3597"
server_key_iv: "5664e350a7f177b927a7b03f095cc8de"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:03.542802 2484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd"
format_stamp: "Formatted at 2026-08-10 23:03:03 on dist-test-slave-6bbx"
server_key: "1811299e44efc02ca959f5ead5cd3597"
server_key_iv: "5664e350a7f177b927a7b03f095cc8de"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:03.550421 2484 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 23:03:03.572032 2508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:03.574033 2484 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260810 23:03:03.574193 2484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd"
format_stamp: "Formatted at 2026-08-10 23:03:03 on dist-test-slave-6bbx"
server_key: "1811299e44efc02ca959f5ead5cd3597"
server_key_iv: "5664e350a7f177b927a7b03f095cc8de"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:03.574343 2484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:03.590415 2484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:03.594518 2484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:03.594831 2484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:03.595789 2484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:03.597100 2484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:03.597232 2484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:03.597368 2484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:03.597440 2484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:03.609710 2484 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:34065
I20260810 23:03:03.609933 2682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:34065 every 8 connection(s)
I20260810 23:03:03.611188 2484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:03.618000 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 2484
I20260810 23:03:03.618156 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:03.618537 15584 external_mini_cluster.cc:1428] Setting key 323b03b46ec5ea068373dfc0ffe71fbd
Aug 10 23:03:03 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402983, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:03:03.629612 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:03.615732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:36449 (local address 127.15.56.62:44421)
0810 23:03:03.615892 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:03.615897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:03.615917 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:03:03.616822 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:03.616827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:03.616893 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:03.616998 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:03.618754 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.619575 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.621533 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.621899 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.622022 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:03.625265 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:03.625269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:03.625272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:03.625305 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:03.626956 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:03.627707 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:03.627711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:03.627713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:03.627812 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:03.628239 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:03.628243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:03.628244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:03.628472 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:03.628564 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:03.629312 (+ 748us) server_negotiation.cc:299] Negotiation successful
0810 23:03:03.629447 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 23:03:03.630905 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:03.614404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44421 (local address 127.15.56.3:36449)
0810 23:03:03.615116 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:03.615193 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:03.616518 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:03.617270 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:03.617282 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:03.617742 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:03.618525 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:03.618550 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.619877 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.619880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:03.621372 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:03.621389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.622151 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.622156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:03.622403 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:03.623172 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:03.623194 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:03.625029 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:03.627095 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:03.627103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:03.627126 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:03.627556 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:03.627907 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:03.627911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:03.627913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:03.628142 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:03.628576 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:03.628584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:03.629000 (+ 416us) client_negotiation.cc:769] Sending connection context
0810 23:03:03.629264 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 23:03:03.629660 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":182,"threads_started":1}
I20260810 23:03:03.632335 2684 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44421
I20260810 23:03:03.632614 2684 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:03.633234 2684 heartbeater.cc:511] Master 127.15.56.62:44421 requested a full tablet report, sending...
I20260810 23:03:03.634596 1947 ts_manager.cc:194] Registered new tserver with Master: 7028b5f58bf44c6cbfef9a46e4da9fdd (127.15.56.3:34065)
I20260810 23:03:03.635382 1947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:36449
I20260810 23:03:03.649022 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:03:03.660779 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:03.651526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.15.56.62:44421)
0810 23:03:03.651652 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:03.651657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:03.651951 (+ 294us) server_negotiation.cc:407] Connection header received
0810 23:03:03.652114 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:03.652118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:03.652176 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:03.652264 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:03.653079 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.653945 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.655115 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.655468 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.655579 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:03.657237 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:03.657242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:03.657244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:03.657283 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:03.658987 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:03.659447 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:03.659451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:03.659453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:03.659545 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:03.659852 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:03.659855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:03.659857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:03.660100 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:03.660243 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:03.660476 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 23:03:03.660606 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:03:03.664624 1947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:03.669227 1947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:03.695786 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:03.682019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:34065 (local address 127.0.0.1:36130)
0810 23:03:03.687521 (+ 5502us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:03.687535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:03.687641 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:03.688578 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:03.688585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:03.688628 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:03.688818 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:03.688828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.691854 (+ 3026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.691858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:03.693611 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:03.693618 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.695077 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.695081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:03.695213 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:03.695276 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 23:03:03.695404 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:03:03.695563 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5432,"mutex_wait_us":1,"spinlock_wait_cycles":4608,"thread_start_us":143,"threads_started":1}
I20260810 23:03:03.696808 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:03.682505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.15.56.3:34065)
0810 23:03:03.687318 (+ 4813us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:03.687334 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:03.687780 (+ 446us) server_negotiation.cc:407] Connection header received
0810 23:03:03.687896 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:03.687905 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:03.688198 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:03.688373 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:03.690505 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.691708 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.693759 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.694689 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.694806 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:03.695004 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:03.696531 (+ 1527us) server_negotiation.cc:299] Negotiation successful
0810 23:03:03.696653 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4656,"thread_start_us":185,"threads_started":1}
I20260810 23:03:03.698916 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:03.681856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:44517 (local address 127.0.0.1:40690)
0810 23:03:03.686852 (+ 4996us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:03.686871 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:03.686991 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:03.689491 (+ 2500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:03.689493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:03.689512 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:03.689612 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:03.689619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.692010 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.692014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:03.694471 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:03.694483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.698505 (+ 4022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.698509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:03.698604 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:03.698627 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:03:03.698695 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:03:03.698774 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4897,"mutex_wait_us":1301,"thread_start_us":108,"threads_started":1}
I20260810 23:03:03.699746 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:03.682505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.15.56.2:44517)
0810 23:03:03.688842 (+ 6337us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:03.688859 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:03.688884 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:03:03.688960 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:03.688965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:03.689184 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:03.689343 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:03.689753 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.691881 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.694652 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.695483 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.698264 (+ 2781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:03.698434 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:03.699461 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0810 23:03:03.699581 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6218,"thread_start_us":71,"threads_started":1}
I20260810 23:03:03.701254 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:03.680531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:45623 (local address 127.0.0.1:58230)
0810 23:03:03.681008 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:03.681068 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:03.681212 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:03.688577 (+ 7365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:03.688585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:03.688631 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:03.689026 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:03.689033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.692224 (+ 3191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.692227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:03.693444 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:03.693457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.700835 (+ 7378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.700840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:03.700934 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:03.700959 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:03:03.701032 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 23:03:03.701106 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":319,"spinlock_wait_cycles":8448,"thread_start_us":133,"threads_started":1}
I20260810 23:03:03.700732 2580 tablet_service.cc:1467] Processing CreateTablet for tablet d4ac8e3f873d4888bb8a27301eb15e26 (DEFAULT_TABLE table=test-table [id=0569adf376c546ae8126e266f8a9a64e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:03.702036 2580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ac8e3f873d4888bb8a27301eb15e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:03.702991 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:03.680662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.15.56.1:45623)
0810 23:03:03.687818 (+ 7156us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:03.687826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:03.687854 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:03:03.687948 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:03.687954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:03.688196 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:03.688372 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:03.689167 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.690352 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.696827 (+ 6475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:03.697675 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:03.697785 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:03.699375 (+ 1590us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:03.702682 (+ 3307us) server_negotiation.cc:299] Negotiation successful
0810 23:03:03.702848 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7031,"thread_start_us":127,"threads_started":1}
I20260810 23:03:03.702939 2373 tablet_service.cc:1467] Processing CreateTablet for tablet d4ac8e3f873d4888bb8a27301eb15e26 (DEFAULT_TABLE table=test-table [id=0569adf376c546ae8126e266f8a9a64e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:03.704164 2373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ac8e3f873d4888bb8a27301eb15e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:03.706000 2169 tablet_service.cc:1467] Processing CreateTablet for tablet d4ac8e3f873d4888bb8a27301eb15e26 (DEFAULT_TABLE table=test-table [id=0569adf376c546ae8126e266f8a9a64e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:03.707144 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ac8e3f873d4888bb8a27301eb15e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:03.724388 2711 tablet_bootstrap.cc:492] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294: Bootstrap starting.
I20260810 23:03:03.727273 2711 tablet_bootstrap.cc:654] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:03.728344 2711 log.cc:824] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:03.729267 2714 tablet_bootstrap.cc:492] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd: Bootstrap starting.
I20260810 23:03:03.732746 2714 tablet_bootstrap.cc:654] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:03.734089 2714 log.cc:824] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:03.735630 2710 tablet_bootstrap.cc:492] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b: Bootstrap starting.
I20260810 23:03:03.738852 2710 tablet_bootstrap.cc:654] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:03.740525 2710 log.cc:824] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:03.743462 2711 tablet_bootstrap.cc:492] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294: No bootstrap required, opened a new log
I20260810 23:03:03.743791 2711 ts_tablet_manager.cc:1397] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20260810 23:03:03.750190 2714 tablet_bootstrap.cc:492] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd: No bootstrap required, opened a new log
I20260810 23:03:03.750620 2714 ts_tablet_manager.cc:1397] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.000s
I20260810 23:03:03.752821 2711 raft_consensus.cc:348] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00660c68ba4d7381f2f1000f865294" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45623 } } peers { permanent_uuid: "74daeff6ac984d30a4033e49dcd06c6b" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44517 } } peers { permanent_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 34065 } }
I20260810 23:03:03.755417 2710 tablet_bootstrap.cc:492] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b: No bootstrap required, opened a new log
I20260810 23:03:03.755349 2714 raft_consensus.cc:348] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00660c68ba4d7381f2f1000f865294" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45623 } } peers { permanent_uuid: "74daeff6ac984d30a4033e49dcd06c6b" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44517 } } peers { permanent_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 34065 } }
I20260810 23:03:03.755589 2711 raft_consensus.cc:374] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:03.755748 2711 raft_consensus.cc:729] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d00660c68ba4d7381f2f1000f865294, State: Initialized, Role: FOLLOWER
I20260810 23:03:03.755782 2714 raft_consensus.cc:374] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:03.755956 2714 raft_consensus.cc:729] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7028b5f58bf44c6cbfef9a46e4da9fdd, State: Initialized, Role: FOLLOWER
I20260810 23:03:03.756398 2711 consensus_queue.cc:260] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00660c68ba4d7381f2f1000f865294" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45623 } } peers { permanent_uuid: "74daeff6ac984d30a4033e49dcd06c6b" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44517 } } peers { permanent_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 34065 } }
I20260810 23:03:03.756527 2714 consensus_queue.cc:260] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00660c68ba4d7381f2f1000f865294" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45623 } } peers { permanent_uuid: "74daeff6ac984d30a4033e49dcd06c6b" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44517 } } peers { permanent_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 34065 } }
I20260810 23:03:03.757913 2714 ts_tablet_manager.cc:1428] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 23:03:03.758397 2684 heartbeater.cc:503] Master 127.15.56.62:44421 was elected leader, sending a full tablet report...
I20260810 23:03:03.759047 2711 ts_tablet_manager.cc:1428] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294: Time spent starting tablet: real 0.015s user 0.001s sys 0.005s
I20260810 23:03:03.759461 2272 heartbeater.cc:503] Master 127.15.56.62:44421 was elected leader, sending a full tablet report...
I20260810 23:03:03.759701 2710 ts_tablet_manager.cc:1397] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.004s
I20260810 23:03:03.766301 2710 raft_consensus.cc:348] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00660c68ba4d7381f2f1000f865294" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45623 } } peers { permanent_uuid: "74daeff6ac984d30a4033e49dcd06c6b" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44517 } } peers { permanent_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 34065 } }
I20260810 23:03:03.766676 2710 raft_consensus.cc:374] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:03.766794 2710 raft_consensus.cc:729] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74daeff6ac984d30a4033e49dcd06c6b, State: Initialized, Role: FOLLOWER
I20260810 23:03:03.767383 2710 consensus_queue.cc:260] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00660c68ba4d7381f2f1000f865294" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45623 } } peers { permanent_uuid: "74daeff6ac984d30a4033e49dcd06c6b" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44517 } } peers { permanent_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 34065 } }
I20260810 23:03:03.768395 2474 heartbeater.cc:503] Master 127.15.56.62:44421 was elected leader, sending a full tablet report...
I20260810 23:03:03.770534 2710 ts_tablet_manager.cc:1428] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
Aug 10 23:03:03 dist-test-slave-6bbx krb5kdc[1247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402982, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
W20260810 23:03:03.851812 2273 tablet.cc:2369] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:03:03.863895 2685 tablet.cc:2369] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:03:03.940573 2475 tablet.cc:2369] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:03:04.078577 2721 raft_consensus.cc:482] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:04.078806 2721 raft_consensus.cc:504] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00660c68ba4d7381f2f1000f865294" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45623 } } peers { permanent_uuid: "74daeff6ac984d30a4033e49dcd06c6b" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44517 } } peers { permanent_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 34065 } }
I20260810 23:03:04.080238 2721 leader_election.cc:288] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d00660c68ba4d7381f2f1000f865294 (127.15.56.1:45623), 74daeff6ac984d30a4033e49dcd06c6b (127.15.56.2:44517)
I20260810 23:03:04.085129 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:04.080979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:34053 (local address 127.15.56.2:44517)
0810 23:03:04.081119 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:04.081124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:04.081310 (+ 186us) server_negotiation.cc:407] Connection header received
0810 23:03:04.081441 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:04.081445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:04.081495 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:04.081569 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:04.081867 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:04.082585 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:04.083730 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:04.084455 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:04.084594 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:04.084684 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:04.084896 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 23:03:04.084985 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:03:04.085152 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:04.080525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:45623 (local address 127.15.56.3:43083)
0810 23:03:04.080683 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:04.080701 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:04.080823 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:04.081248 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:04.081253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:04.081277 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:04.081408 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:04.081416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:04.082444 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:04.082448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:04.083720 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:04.083733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:04.084776 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:04.084779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:04.084872 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:04.084895 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:03:04.084963 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:03:04.085038 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":31}
I20260810 23:03:04.085177 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:04.080856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:44517 (local address 127.15.56.3:34053)
0810 23:03:04.081201 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:04.081213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:04.081335 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:04.081597 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:04.081600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:04.081620 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:04.081745 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:04.081752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:04.082701 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:04.082705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:04.083584 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:04.083596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:04.084624 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:04.084628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:04.084744 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:04.084776 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:03:04.084861 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:03:04.084951 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":23,"thread_start_us":101,"threads_started":1}
I20260810 23:03:04.085966 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:04.080651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:43083 (local address 127.15.56.1:45623)
0810 23:03:04.080787 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:04.080792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:04.080808 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:03:04.080963 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:04.080966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:04.081018 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:04.081098 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:04.081552 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:04.082310 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:04.083955 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:04.084555 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:04.084647 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:04.084705 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:04.085778 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0810 23:03:04.085865 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:03:04.086139 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ac8e3f873d4888bb8a27301eb15e26" candidate_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74daeff6ac984d30a4033e49dcd06c6b" is_pre_election: true
I20260810 23:03:04.086536 2181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ac8e3f873d4888bb8a27301eb15e26" candidate_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00660c68ba4d7381f2f1000f865294" is_pre_election: true
I20260810 23:03:04.086567 2407 raft_consensus.cc:2454] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7028b5f58bf44c6cbfef9a46e4da9fdd in term 0.
I20260810 23:03:04.086843 2181 raft_consensus.cc:2454] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7028b5f58bf44c6cbfef9a46e4da9fdd in term 0.
I20260810 23:03:04.087198 2511 leader_election.cc:302] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7028b5f58bf44c6cbfef9a46e4da9fdd, 74daeff6ac984d30a4033e49dcd06c6b; no voters:
I20260810 23:03:04.087497 2721 raft_consensus.cc:2781] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:04.087601 2721 raft_consensus.cc:482] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:04.087654 2721 raft_consensus.cc:3037] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:04.088860 2721 raft_consensus.cc:504] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00660c68ba4d7381f2f1000f865294" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45623 } } peers { permanent_uuid: "74daeff6ac984d30a4033e49dcd06c6b" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44517 } } peers { permanent_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 34065 } }
I20260810 23:03:04.089226 2721 leader_election.cc:288] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [CANDIDATE]: Term 1 election: Requested vote from peers 1d00660c68ba4d7381f2f1000f865294 (127.15.56.1:45623), 74daeff6ac984d30a4033e49dcd06c6b (127.15.56.2:44517)
I20260810 23:03:04.089685 2407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ac8e3f873d4888bb8a27301eb15e26" candidate_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74daeff6ac984d30a4033e49dcd06c6b"
I20260810 23:03:04.089685 2181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ac8e3f873d4888bb8a27301eb15e26" candidate_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d00660c68ba4d7381f2f1000f865294"
I20260810 23:03:04.089820 2181 raft_consensus.cc:3037] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:04.089820 2407 raft_consensus.cc:3037] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:04.091087 2181 raft_consensus.cc:2454] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7028b5f58bf44c6cbfef9a46e4da9fdd in term 1.
I20260810 23:03:04.091087 2407 raft_consensus.cc:2454] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7028b5f58bf44c6cbfef9a46e4da9fdd in term 1.
I20260810 23:03:04.091456 2511 leader_election.cc:302] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d00660c68ba4d7381f2f1000f865294, 7028b5f58bf44c6cbfef9a46e4da9fdd; no voters:
I20260810 23:03:04.091698 2721 raft_consensus.cc:2781] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:04.091948 2721 raft_consensus.cc:686] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 1 LEADER]: Becoming Leader. State: Replica: 7028b5f58bf44c6cbfef9a46e4da9fdd, State: Running, Role: LEADER
I20260810 23:03:04.092329 2721 consensus_queue.cc:237] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00660c68ba4d7381f2f1000f865294" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45623 } } peers { permanent_uuid: "74daeff6ac984d30a4033e49dcd06c6b" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44517 } } peers { permanent_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 34065 } }
I20260810 23:03:04.096110 1949 catalog_manager.cc:5495] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7028b5f58bf44c6cbfef9a46e4da9fdd (127.15.56.3). New cstate: current_term: 1 leader_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d00660c68ba4d7381f2f1000f865294" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74daeff6ac984d30a4033e49dcd06c6b" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7028b5f58bf44c6cbfef9a46e4da9fdd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 34065 } health_report { overall_health: HEALTHY } } }
I20260810 23:03:04.145368 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:04.141208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.15.56.3:34065)
0810 23:03:04.141379 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:04.141384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:04.141401 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:03:04.141498 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:04.141501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:04.141581 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:04.141701 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:04.141925 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:04.142709 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:04.143541 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:04.143881 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:04.143983 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:04.144266 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:04.144904 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:04.145048 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:04.145095 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 23:03:04.145168 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:03:04.152576 2181 raft_consensus.cc:1261] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294 [term 1 FOLLOWER]: Refusing update from remote peer 7028b5f58bf44c6cbfef9a46e4da9fdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:04.153393 2741 consensus_queue.cc:1035] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d00660c68ba4d7381f2f1000f865294" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 45623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:04.153510 2407 raft_consensus.cc:1261] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b [term 1 FOLLOWER]: Refusing update from remote peer 7028b5f58bf44c6cbfef9a46e4da9fdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:04.154258 2741 consensus_queue.cc:1035] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "74daeff6ac984d30a4033e49dcd06c6b" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:04.166774 2750 mvcc.cc:205] Tried to move back new op lower bound from 7317106623080509440 to 7317106622845251584. Current Snapshot: MvccSnapshot[applied={T|T < 7317106623080509440}]
I20260810 23:03:04.169699 2748 mvcc.cc:205] Tried to move back new op lower bound from 7317106623080509440 to 7317106622845251584. Current Snapshot: MvccSnapshot[applied={T|T < 7317106623080509440}]
I20260810 23:03:04.174326 1947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:04.174543 1947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:04.177467 1947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19faab31330d4664b93d9e79a3a3a3ca: Sending DeleteTablet for 3 replicas of tablet d4ac8e3f873d4888bb8a27301eb15e26
I20260810 23:03:04.178661 2169 tablet_service.cc:1514] Processing DeleteTablet for tablet d4ac8e3f873d4888bb8a27301eb15e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:04 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:58230
I20260810 23:03:04.178659 2580 tablet_service.cc:1514] Processing DeleteTablet for tablet d4ac8e3f873d4888bb8a27301eb15e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:04 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:36130
I20260810 23:03:04.179276 2763 tablet_replica.cc:330] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294: stopping tablet replica
I20260810 23:03:04.179451 2373 tablet_service.cc:1514] Processing DeleteTablet for tablet d4ac8e3f873d4888bb8a27301eb15e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:04 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:40690
I20260810 23:03:04.179790 2763 raft_consensus.cc:2229] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:04.180105 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 2070
I20260810 23:03:04.180132 2765 tablet_replica.cc:330] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b: stopping tablet replica
I20260810 23:03:04.180172 2763 raft_consensus.cc:2258] T d4ac8e3f873d4888bb8a27301eb15e26 P 1d00660c68ba4d7381f2f1000f865294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:04.186697 2765 raft_consensus.cc:2229] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:04.187170 2765 raft_consensus.cc:2258] T d4ac8e3f873d4888bb8a27301eb15e26 P 74daeff6ac984d30a4033e49dcd06c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:04.193257 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 2279
W20260810 23:03:04.193956 1926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:45623 (error 108)
I20260810 23:03:04.209055 2764 tablet_replica.cc:330] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd: stopping tablet replica
I20260810 23:03:04.209613 2764 raft_consensus.cc:2229] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:04.210269 2764 raft_consensus.cc:2258] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:04.211048 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:04.208494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:45623 (local address 127.0.0.1:58240)
0810 23:03:04.210793 (+ 2299us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:04.210914 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:45623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2152,"thread_start_us":85,"threads_started":1}
W20260810 23:03:04.211336 1926 catalog_manager.cc:4584] TS 1d00660c68ba4d7381f2f1000f865294 (127.15.56.1:45623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4ac8e3f873d4888bb8a27301eb15e26: Network error: Client connection negotiation failed: client connection to 127.15.56.1:45623: connect: Connection refused (error 111)
I20260810 23:03:04.212919 2764 ts_tablet_manager.cc:1905] T d4ac8e3f873d4888bb8a27301eb15e26 P 7028b5f58bf44c6cbfef9a46e4da9fdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:04.213413 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 2484
W20260810 23:03:04.223465 2767 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:04.212724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:44517 (local address 127.0.0.1:40692)
0810 23:03:04.223209 (+ 10485us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:04.223309 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.2:44517: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":10335}
I20260810 23:03:04.237645 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 1900
I20260810 23:03:04.256076 15584 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:03:04.601923 15584 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:03:04.602085 15584 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:03:04.958649 15584 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:03:04.958 UTC [1287] LOG: received smart shutdown request
2026-08-10 23:03:04.962 UTC [1287] LOG: background worker "logical replication launcher" (PID 1301) exited with exit code 1
2026-08-10 23:03:04.963 UTC [1292] LOG: shutting down
2026-08-10 23:03:05.195 UTC [1287] LOG: database system is shut down
2026-08-10T23:03:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50697 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:05 dist-test-slave-6bbx krb5kdc[2781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:05 dist-test-slave-6bbx krb5kdc[2781](info): set up 2 sockets
Aug 10 23:03:05 dist-test-slave-6bbx krb5kdc[2781](info): commencing operation
krb5kdc: starting...
W20260810 23:03:06.465850 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:03:06 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/rangeradmin_127.15.56.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/rangeradmin_127.15.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/rangerlookup_127.15.56.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/rangerlookup_127.15.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/HTTP_127.15.56.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/HTTP_127.15.56.20@KRBTEST.COM.keytab.
I20260810 23:03:06.658442 15584 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWzUZ3m/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/postgres -l logfile start
2026-08-10 23:03:09.171 UTC [2824] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:03:09.172 UTC [2824] LOG: listening on IPv4 address "127.15.56.20", port 53123
2026-08-10 23:03:09.175 UTC [2824] LOG: listening on Unix socket "/tmp/.s.PGSQL.53123"
2026-08-10 23:03:09.195 UTC [2827] LOG: database system was shut down at 2026-08-10 23:03:07 UTC
2026-08-10 23:03:09.199 UTC [2824] LOG: database system is ready to accept connections
I20260810 23:03:10.284860 15584 mini_postgres.cc:96] Postgres bound to 53123
2026-08-10 23:03:10.292 UTC [2838] FATAL: database "slave" does not exist
127.15.56.20:53123 - accepting connections
I20260810 23:03:10.293292 15584 mini_ranger.cc:161] Starting Ranger...
I20260810 23:03:10.309085 15584 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:03:10.723729 15584 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:03:10.723816 15584 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-admin
2026-08-10 23:03:11,010 [I] DB FLAVOR :POSTGRES
2026-08-10 23:03:11,010 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:03:11,010 [I] Checking connection..
2026-08-10 23:03:11,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:03:11,280 [I] Checking connection passed.
2026-08-10 23:03:11,280 [I] --------- Verifying version history table ---------
2026-08-10 23:03:11,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:03:11,559 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:03:11,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:03:11,839 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:03:11,840 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:03:11,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:03:12.079 UTC [2951] WARNING: there is no transaction in progress
2026-08-10 23:03:12,088 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:03:12,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:03:12,335 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:03:12,335 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:03:12,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:03:12,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:03:12,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6bbx.c.gcp-upstream.internal','N') ;"
2026-08-10 23:03:13,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:03:13,349 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:03:13,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:03:13,623 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:03:13,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:03:13,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:03:14,145 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:03:14,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:03:15.128 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.146 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.162 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.178 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.196 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.216 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.275 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.282 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.298 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.309 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.319 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.335 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.345 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.352 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.359 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:15.366 UTC [3115] WARNING: there is no transaction in progress
2026-08-10 23:03:16,191 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:03:16,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:03:16,459 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:03:16,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:03:16,718 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:03:16,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:03:16,992 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:03:16,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:03:17,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:03:17,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6bbx.c.gcp-upstream.internal';"
2026-08-10 23:03:17,762 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:03:17,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:03:18,032 [I] DB_PATCHES have already been applied
I20260810 23:03:18.039220 15584 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWzUZ3m/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:03:18.039306 15584 mini_ranger.cc:193] Using host: 127.15.56.20
I20260810 23:03:18.044519 15584 mini_ranger.cc:227] Ranger admin URL: http://127.15.56.20:48983
Aug 10, 2026 11:03:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:03:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:03:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:03:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:03:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:03:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:03:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:03:19.308470 15584 mini_ranger.cc:160] Time spent starting Ranger: real 9.015s user 0.004s sys 0.008s
Aug 10, 2026 11:03:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:03:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 11:03:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:03:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:03:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:03:41.950320 15584 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.56.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.56.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/rangerkms_127.15.56.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.56.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/rangerkms_127.15.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.56.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.56.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.56.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/HTTP_127.15.56.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.56.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/HTTP_127.15.56.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:03:42 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:03:42.046880 15584 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:03:42.058888 15584 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:03:42.915030 15584 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:03:42.915122 15584 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-kms
I20260810 23:03:42.915140 15584 mini_ranger_kms.cc:227] Using postgres at 127.15.56.20:53123
2026-08-10 23:03:43,097 [I] DB FLAVOR :POSTGRES
2026-08-10 23:03:43,098 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:03:43,098 [I] Checking connection
2026-08-10 23:03:43,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:03:43,346 [I] connection success
2026-08-10 23:03:43,347 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:03:43,347 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:03:43,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:03:43,598 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:03:43,599 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:03:43,599 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:03:43,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.56.20:53123/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:03:43,869 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:03:44.361920 15584 mini_ranger_kms.cc:310] Created kms service
I20260810 23:03:44.572870 15584 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:03:44.621598 15584 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:03:44 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:45.097800 15584 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:03:45.097975 15584 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWzUZ3m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWzUZ3m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:03:45.097998 15584 mini_ranger_kms.cc:240] Using host: 127.15.56.20
I20260810 23:03:45.103000 15584 mini_ranger_kms.cc:276] Ranger KMS PID: 3401
I20260810 23:03:45.103117 15584 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.56.20:34445
Aug 10, 2026 11:03:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:03:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:03:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:03:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:03:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:03:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 23:03:47 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403027, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.56.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:47.627913 15584 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.581s user 0.012s sys 0.151s
I20260810 23:03:47.628038 15584 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:03:47.628103 15584 mini_ranger_kms.cc:401] 127.15.56.20:34445/kms/v1/keys
Aug 10 23:03:47 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403027, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.56.20@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
Aug 10 23:03:48 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403027, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.56.20@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
Aug 10 23:03:50 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:03:50.687098 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33479
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:33479
--ranger_config_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:34445/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:50.810863 3474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:50.811187 3474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:50.811230 3474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:50.815516 3474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:50.815594 3474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:50.815613 3474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:50.815627 3474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:50.815641 3474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:50.821694 3474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.56.20:34445/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:33479
--ranger_config_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:50.822997 3474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:50.824292 3474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:50.834151 3483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:50.834213 3474 server_base.cc:1034] running on GCE node
W20260810 23:03:50.834151 3490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:50.834242 3488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:50.835266 3474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:50.836725 3474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:50.837899 3474 hybrid_clock.cc:648] HybridClock initialized: now 1786403030837869 us; error 55 us; skew 500 ppm
Aug 10 23:03:50 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:50.841226 3474 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:03:50.842298 3474 webserver.cc:460] Webserver started at http://127.15.56.62:44569/ using document root <none> and password file <none>
I20260810 23:03:50.842873 3474 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:50.842936 3474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:50.843149 3474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:03:50 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:50.911780 3474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "5afa7dd02923413b8162ea41990e263a"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-6bbx"
server_key: "ed0bdea4cbb54dcac51b1ae05b762b26"
server_key_iv: "3ff6af793c97a6e1ce458dae849b8916"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:50.912464 3474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "5afa7dd02923413b8162ea41990e263a"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-6bbx"
server_key: "ed0bdea4cbb54dcac51b1ae05b762b26"
server_key_iv: "3ff6af793c97a6e1ce458dae849b8916"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:50.916723 3474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:03:50.935650 3501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.937471 3474 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:03:50.937650 3474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "5afa7dd02923413b8162ea41990e263a"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-6bbx"
server_key: "ed0bdea4cbb54dcac51b1ae05b762b26"
server_key_iv: "3ff6af793c97a6e1ce458dae849b8916"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:50.937805 3474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:50.952997 3474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:50.956637 3474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:50.956840 3474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:50.965200 3474 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:33479
I20260810 23:03:50.965221 3585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:33479 every 8 connection(s)
I20260810 23:03:50.966396 3474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:50.966807 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 3474
I20260810 23:03:50.966964 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:03:50.967244 15584 external_mini_cluster.cc:1428] Setting key c721f48ee19f67e0ef3130ca715c010c
I20260810 23:03:50.971547 3587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 23:03:50 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:03:50.979235 3587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a: Bootstrap starting.
I20260810 23:03:50.981364 3590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.968719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.15.56.62:33479)
0810 23:03:50.969215 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:50.969232 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:50.969274 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:03:50.969949 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:50.969968 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:50.970350 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:50.970702 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:50.971580 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.972698 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.973618 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.974132 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.974287 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.976855 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:50.976861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:50.976876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:50.976908 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:50.978452 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:50.978897 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:50.978902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:50.978907 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:50.979017 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:50.979285 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:50.979288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:50.979290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:50.979800 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:50.979994 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:50.980289 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 23:03:50.980593 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":149,"threads_started":1}
I20260810 23:03:50.982203 3587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:50.983604 3587 log.cc:824] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:50.986172 3587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a: No bootstrap required, opened a new log
I20260810 23:03:50.990561 3587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } }
I20260810 23:03:50.990969 3587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:50.991072 3587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afa7dd02923413b8162ea41990e263a, State: Initialized, Role: FOLLOWER
I20260810 23:03:50.991633 3587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } }
I20260810 23:03:50.991844 3587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:50.991916 3587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:50.992019 3587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:50.993326 3587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } }
I20260810 23:03:50.993782 3587 leader_election.cc:302] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5afa7dd02923413b8162ea41990e263a; no voters:
I20260810 23:03:50.994221 3587 leader_election.cc:288] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:50.994338 3592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:50.994580 3592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 1 LEADER]: Becoming Leader. State: Replica: 5afa7dd02923413b8162ea41990e263a, State: Running, Role: LEADER
I20260810 23:03:50.995031 3592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } }
I20260810 23:03:50.995775 3587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:50.997200 3592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5afa7dd02923413b8162ea41990e263a. Latest consensus state: current_term: 1 leader_uuid: "5afa7dd02923413b8162ea41990e263a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } } }
I20260810 23:03:50.997429 3592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:50.997668 3594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5afa7dd02923413b8162ea41990e263a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } } }
I20260810 23:03:50.997761 3594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:50.997924 3611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:50.998206 3587 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:03:51.001545 3611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:51.011770 3611 catalog_manager.cc:1347] Generated new cluster ID: 5dad87a8dca9424c9bb1ecc009904c7a
I20260810 23:03:51.011876 3611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:51.024457 3611 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:51.026043 3611 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:51.034631 3611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a: Generated new TSK 0
I20260810 23:03:51.035848 3611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:03:51.804211 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:34445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33479
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:51.973332 3653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:51.973825 3653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:51.973941 3653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:51.979979 3653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:51.980058 3653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:51.980142 3653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:03:51.985857 3653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.56.20:34445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:51.987260 3653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:51.988854 3653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:52.001969 3666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:52.001968 3664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:52.002625 3668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:52.003042 3653 server_base.cc:1034] running on GCE node
I20260810 23:03:52.003587 3653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:52.004524 3653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:52.005739 3653 hybrid_clock.cc:648] HybridClock initialized: now 1786403032005721 us; error 40 us; skew 500 ppm
Aug 10 23:03:52 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:52.009516 3653 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:03:52.010701 3653 webserver.cc:460] Webserver started at http://127.15.56.1:35839/ using document root <none> and password file <none>
I20260810 23:03:52.011548 3653 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:52.011639 3653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:52.011932 3653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:03:52 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:52.028723 3653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "6f19d1872ab042a3a9d4ead5843ec89d"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "5c23f4d4e55f868b0e29e2f90fd7dee7"
server_key_iv: "15577f37b3b0faa5e9aa61ab0d783bf0"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:52.029350 3653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "6f19d1872ab042a3a9d4ead5843ec89d"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "5c23f4d4e55f868b0e29e2f90fd7dee7"
server_key_iv: "15577f37b3b0faa5e9aa61ab0d783bf0"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:52.044970 3653 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.005s sys 0.000s
I20260810 23:03:52.068041 3676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:52.070099 3653 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260810 23:03:52.070269 3653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "6f19d1872ab042a3a9d4ead5843ec89d"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "5c23f4d4e55f868b0e29e2f90fd7dee7"
server_key_iv: "15577f37b3b0faa5e9aa61ab0d783bf0"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:52.070426 3653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:52.094456 3653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:52.098557 3653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:52.098803 3653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:52.099618 3653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:52.100876 3653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:52.100950 3653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:52.101011 3653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:52.101094 3653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:52.113178 3653 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:41567
I20260810 23:03:52.113240 3853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:41567 every 8 connection(s)
I20260810 23:03:52.114676 3653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:52.116673 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 3653
I20260810 23:03:52.116812 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:52.117102 15584 external_mini_cluster.cc:1428] Setting key 7609defecf75aca12403c8d325fdf4cd
Aug 10 23:03:52 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:03:52.132848 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:52.117065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:35777 (local address 127.15.56.62:33479)
0810 23:03:52.120154 (+ 3089us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:52.120159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:52.120175 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:03:52.120216 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:52.120219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:52.120268 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:52.121855 (+ 1587us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:52.121980 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.122825 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.124595 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.124969 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.125044 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.128233 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:52.128238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:52.128241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:52.128271 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:52.130155 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:52.130822 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:52.130826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:52.130828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:52.130922 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:52.131359 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:52.131363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:52.131365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:52.131599 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:52.131710 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:52.132493 (+ 783us) server_negotiation.cc:299] Negotiation successful
0810 23:03:52.132641 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3001,"thread_start_us":90,"threads_started":1}
I20260810 23:03:52.134850 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:52.117634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33479 (local address 127.15.56.1:35777)
0810 23:03:52.118262 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:52.118324 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:52.119396 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:52.120601 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:52.120612 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:52.120988 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:52.121705 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.121725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.122971 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.122975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.124445 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.124458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.125272 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.125276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.125492 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.126213 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:52.126236 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:52.128007 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:52.130324 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:52.130330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:52.130348 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:52.130703 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:52.131033 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:52.131036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:52.131038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:52.131249 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:52.131784 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:52.131790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:52.132168 (+ 378us) client_negotiation.cc:769] Sending connection context
0810 23:03:52.133245 (+ 1077us) client_negotiation.cc:240] Negotiation successful
0810 23:03:52.133631 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":153,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
I20260810 23:03:52.136831 3854 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33479
I20260810 23:03:52.137176 3854 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:52.137939 3854 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
I20260810 23:03:52.140162 3517 ts_manager.cc:194] Registered new tserver with Master: 6f19d1872ab042a3a9d4ead5843ec89d (127.15.56.1:41567)
I20260810 23:03:52.141817 3517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:35777
Aug 10 23:03:52 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:03:52.159827 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:34445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33479
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:52.327121 3861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:52.327533 3861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:52.327596 3861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:52.334015 3861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:52.334162 3861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:52.334275 3861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:03:52.342679 3861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.56.20:34445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:52.344323 3861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:52.345786 3861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:52.362174 3875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:52.362699 3873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:52.363643 3861 server_base.cc:1034] running on GCE node
W20260810 23:03:52.364061 3872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:52.364696 3861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:52.365634 3861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:52.369021 3861 hybrid_clock.cc:648] HybridClock initialized: now 1786403032368875 us; error 123 us; skew 500 ppm
Aug 10 23:03:52 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:52.373446 3861 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:03:52.374822 3861 webserver.cc:460] Webserver started at http://127.15.56.2:38847/ using document root <none> and password file <none>
I20260810 23:03:52.375794 3861 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:52.375905 3861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:52.376271 3861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:03:52 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:52.397716 3861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "751ab888f49b42ebbba0dd6bae3e8842"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "7542e890ed2776e7dff086692360a4b5"
server_key_iv: "c7b41f95851e5f949b25f8d7982a8a12"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:52.398576 3861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "751ab888f49b42ebbba0dd6bae3e8842"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "7542e890ed2776e7dff086692360a4b5"
server_key_iv: "c7b41f95851e5f949b25f8d7982a8a12"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:52.403983 3861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 23:03:52.424144 3886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:52.425941 3861 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 23:03:52.426187 3861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "751ab888f49b42ebbba0dd6bae3e8842"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "7542e890ed2776e7dff086692360a4b5"
server_key_iv: "c7b41f95851e5f949b25f8d7982a8a12"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:52.426409 3861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:52.444108 3861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:52.452708 3861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:52.453025 3861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:52.453933 3861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:52.455240 3861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:52.455361 3861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:52.455474 3861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:52.455549 3861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:52.466921 3861 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:39101
I20260810 23:03:52.468407 3861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:52.472646 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 3861
I20260810 23:03:52.472817 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:52.473152 15584 external_mini_cluster.cc:1428] Setting key 5f68c2bac70d5ccdf5daac43094a8e9f
I20260810 23:03:52.483793 4056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:39101 every 8 connection(s)
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
Aug 10 23:03:52 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
I20260810 23:03:52.509541 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:52.488613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:46945 (local address 127.15.56.62:33479)
0810 23:03:52.488798 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:52.488804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:52.489290 (+ 486us) server_negotiation.cc:407] Connection header received
0810 23:03:52.490635 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:52.490639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:52.490698 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:52.490751 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:52.492623 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.493479 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.495298 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.495691 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.495786 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.504561 (+ 8775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:52.504566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:52.504569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:52.504612 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:52.506534 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:52.507380 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:52.507386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:52.507388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:52.507492 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:52.507993 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:52.507996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:52.507998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:52.508268 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:52.508358 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:52.509199 (+ 841us) server_negotiation.cc:299] Negotiation successful
0810 23:03:52.509355 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:03:52.511786 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:52.487166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33479 (local address 127.15.56.2:46945)
0810 23:03:52.489085 (+ 1919us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:52.489161 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:52.490348 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:52.491217 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:52.491228 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:52.491624 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:52.492420 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.492443 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.493662 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.493665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.495139 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.495152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.496039 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.496043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.496285 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.497016 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:52.497040 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:52.504238 (+ 7198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:52.506738 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:52.506748 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:52.506774 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:52.507241 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:52.507608 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:52.507611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:52.507613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:52.507873 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:52.508433 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:52.508441 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:52.508864 (+ 423us) client_negotiation.cc:769] Sending connection context
0810 23:03:52.510066 (+ 1202us) client_negotiation.cc:240] Negotiation successful
0810 23:03:52.510515 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1667,"thread_start_us":232,"threads_started":1}
I20260810 23:03:52.513610 4057 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33479
I20260810 23:03:52.513902 4057 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:52.514573 4057 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
I20260810 23:03:52.515944 3517 ts_manager.cc:194] Registered new tserver with Master: 751ab888f49b42ebbba0dd6bae3e8842 (127.15.56.2:39101)
I20260810 23:03:52.516748 3517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:46945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:03:52.545221 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:34445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33479
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:03:52 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
W20260810 23:03:52.735944 4065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:52.736374 4065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:52.736444 4065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:52.742944 4065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:52.743165 4065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:52.743316 4065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:03:52.751401 4065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.56.20:34445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:52.753149 4065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:52.754596 4065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:52.769897 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:52.770697 4074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:52.772010 4065 server_base.cc:1034] running on GCE node
W20260810 23:03:52.773049 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:52.773653 4065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:52.774605 4065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:52.782013 4065 hybrid_clock.cc:648] HybridClock initialized: now 1786403032781918 us; error 110 us; skew 500 ppm
Aug 10 23:03:52 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:52.786432 4065 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:03:52.787876 4065 webserver.cc:460] Webserver started at http://127.15.56.3:35121/ using document root <none> and password file <none>
I20260810 23:03:52.788666 4065 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:52.788762 4065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:52.789109 4065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:03:52 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:52.805477 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "f03ca5475323412fb8883fa2eaf55c52"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "52164a26e253554adda4dc5b511d0054"
server_key_iv: "a3464136f6d5bbfd1c0fa975999962e5"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:52.806195 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "f03ca5475323412fb8883fa2eaf55c52"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "52164a26e253554adda4dc5b511d0054"
server_key_iv: "a3464136f6d5bbfd1c0fa975999962e5"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:52.811508 4065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 23:03:52.831768 4084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:52.834048 4065 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260810 23:03:52.834271 4065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "f03ca5475323412fb8883fa2eaf55c52"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "52164a26e253554adda4dc5b511d0054"
server_key_iv: "a3464136f6d5bbfd1c0fa975999962e5"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:52.834462 4065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:52.869530 4065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:52.874084 4065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:52.874316 4065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:52.875110 4065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:52.876406 4065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:52.876478 4065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:52.876539 4065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:52.876569 4065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:52.889487 4065 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:42217
I20260810 23:03:52.890916 4065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:52.895051 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 4065
I20260810 23:03:52.895198 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:52.895553 15584 external_mini_cluster.cc:1428] Setting key 783c600cc8797f60f78ef6717b372a7e
I20260810 23:03:52.900183 4261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:42217 every 8 connection(s)
Aug 10 23:03:52 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403032, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:03:52.930213 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:52.902339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:58207 (local address 127.15.56.62:33479)
0810 23:03:52.902478 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:52.902482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:52.908359 (+ 5877us) server_negotiation.cc:407] Connection header received
0810 23:03:52.911912 (+ 3553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:52.911916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:52.911980 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:52.912032 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:52.914006 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.914780 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.916652 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.917006 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.917090 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.923072 (+ 5982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:52.923076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:52.923080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:52.923120 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:52.926966 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:52.927978 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:52.927983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:52.927985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:52.928097 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:52.928594 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:52.928598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:52.928600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:52.928892 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:52.928980 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:52.929859 (+ 879us) server_negotiation.cc:299] Negotiation successful
0810 23:03:52.930011 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:03:52.932597 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:52.902984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33479 (local address 127.15.56.3:58207)
0810 23:03:52.908121 (+ 5137us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:52.908205 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:52.911569 (+ 3364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:52.912545 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:52.912556 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:52.913011 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:52.913811 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.913836 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.914963 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.914966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.916492 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.916505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.917337 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.917340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.917586 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.918369 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:52.918393 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:52.920434 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:52.927208 (+ 6774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:52.927216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:52.927243 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:52.927831 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:52.928234 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:52.928237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:52.928239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:52.928473 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:52.929095 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:52.929103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:52.929520 (+ 417us) client_negotiation.cc:769] Sending connection context
0810 23:03:52.930749 (+ 1229us) client_negotiation.cc:240] Negotiation successful
0810 23:03:52.931191 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4826,"thread_start_us":207,"threads_started":1}
I20260810 23:03:52.936630 4263 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33479
I20260810 23:03:52.936949 4263 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:52.937656 4263 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
I20260810 23:03:52.939163 3517 ts_manager.cc:194] Registered new tserver with Master: f03ca5475323412fb8883fa2eaf55c52 (127.15.56.3:42217)
I20260810 23:03:52.940142 3517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:58207
I20260810 23:03:52.947271 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:03:52.948184 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 3653
I20260810 23:03:53.049705 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 3861
I20260810 23:03:53.068404 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 4065
I20260810 23:03:53.094594 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 3474
I20260810 23:03:53.114415 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33479
--webserver_interface=127.15.56.62
--webserver_port=44569
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:33479
--ranger_config_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:34445/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.56.20:34445/kms with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:53.234388 4268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:53.234694 4268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:53.234735 4268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:53.238914 4268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:53.238998 4268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:53.239017 4268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:53.239032 4268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:53.239046 4268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:53.245263 4268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.56.20:34445/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:33479
--ranger_config_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=44569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:53.246510 4268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:53.247685 4268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:03:53.257413 4268 server_base.cc:1034] running on GCE node
W20260810 23:03:53.257356 4276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:53.257356 4279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:53.257573 4275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:53.258196 4268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:53.259451 4268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:53.260620 4268 hybrid_clock.cc:648] HybridClock initialized: now 1786403033260618 us; error 90 us; skew 500 ppm
Aug 10 23:03:53 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403033, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:53.263702 4268 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:03:53.264741 4268 webserver.cc:460] Webserver started at http://127.15.56.62:44569/ using document root <none> and password file <none>
I20260810 23:03:53.265305 4268 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:53.265377 4268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:03:53 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403033, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:53.345026 4268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 23:03:53.347375 4289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:53.348832 4268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:03:53.348977 4268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "5afa7dd02923413b8162ea41990e263a"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-6bbx"
server_key: "ed0bdea4cbb54dcac51b1ae05b762b26"
server_key_iv: "3ff6af793c97a6e1ce458dae849b8916"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:53.349573 4268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:53.361994 4268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:53.365219 4268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:53.365428 4268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:53.373332 4268 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:33479
I20260810 23:03:53.373350 4363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:33479 every 8 connection(s)
I20260810 23:03:53.374559 4268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:53.379223 4364 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:03:53.381400 4364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a: Bootstrap starting.
I20260810 23:03:53.384663 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 4268
I20260810 23:03:53.385210 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:34445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:41567
--local_ip_for_outbound_sockets=127.15.56.1
--tserver_master_addrs=127.15.56.62:33479
--webserver_port=35839
--webserver_interface=127.15.56.1
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.56.20:34445/kms with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:03:53.399886 4364 log.cc:824] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:53.407089 4364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:03:53.407500 4364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a: Bootstrap complete.
I20260810 23:03:53.411754 4364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } }
I20260810 23:03:53.412163 4364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5afa7dd02923413b8162ea41990e263a, State: Initialized, Role: FOLLOWER
I20260810 23:03:53.412688 4364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } }
I20260810 23:03:53.412819 4364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:53.412886 4364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:53.412983 4364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:03:53.414873 4364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } }
I20260810 23:03:53.415191 4364 leader_election.cc:302] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5afa7dd02923413b8162ea41990e263a; no voters:
I20260810 23:03:53.415525 4364 leader_election.cc:288] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:03:53.415627 4371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:03:53.415884 4371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [term 2 LEADER]: Becoming Leader. State: Replica: 5afa7dd02923413b8162ea41990e263a, State: Running, Role: LEADER
I20260810 23:03:53.416188 4371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } }
I20260810 23:03:53.416833 4364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:53.417565 4372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5afa7dd02923413b8162ea41990e263a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } } }
I20260810 23:03:53.417877 4372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:53.417784 4371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5afa7dd02923413b8162ea41990e263a. Latest consensus state: current_term: 2 leader_uuid: "5afa7dd02923413b8162ea41990e263a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5afa7dd02923413b8162ea41990e263a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33479 } } }
I20260810 23:03:53.418227 4371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:53.418995 4384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:53.419152 4364 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:03:53.425791 4384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:53.426750 4384 catalog_manager.cc:1259] Loaded cluster ID: 5dad87a8dca9424c9bb1ecc009904c7a
I20260810 23:03:53.426831 4384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:53.429877 4384 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:53.430866 4384 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a: Loaded TSK: 0
I20260810 23:03:53.432253 4384 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:03:53.533737 4368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:53.534181 4368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:53.534253 4368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:53.540881 4368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:53.540997 4368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:53.541090 4368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:03:53.549734 4368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.56.20:34445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:41567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=35839
--tserver_master_addrs=127.15.56.62:33479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:53.551445 4368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:53.553082 4368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:53.567082 4424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:53.567368 4428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:53.567641 4425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:53.567739 4368 server_base.cc:1034] running on GCE node
I20260810 23:03:53.568181 4368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:53.568852 4368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:53.572115 4368 hybrid_clock.cc:648] HybridClock initialized: now 1786403033572026 us; error 94 us; skew 500 ppm
Aug 10 23:03:53 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403033, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:53.575387 4368 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:03:53.576493 4368 webserver.cc:460] Webserver started at http://127.15.56.1:35839/ using document root <none> and password file <none>
I20260810 23:03:53.577159 4368 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:53.577256 4368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:03:53 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403033, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:53.605538 4368 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:03:53.608690 4444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:53.610217 4368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:03:53.610374 4368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "6f19d1872ab042a3a9d4ead5843ec89d"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "5c23f4d4e55f868b0e29e2f90fd7dee7"
server_key_iv: "15577f37b3b0faa5e9aa61ab0d783bf0"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:53.610996 4368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:53.626446 4368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:53.629796 4368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:53.630012 4368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:53.630721 4368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:53.631708 4368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:53.631769 4368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:53.631834 4368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:53.631863 4368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:53.642572 4368 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:41567
I20260810 23:03:53.642836 4625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:41567 every 8 connection(s)
I20260810 23:03:53.643882 4368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:53.646116 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 4368
I20260810 23:03:53.646631 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:34445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:39101
--local_ip_for_outbound_sockets=127.15.56.2
--tserver_master_addrs=127.15.56.62:33479
--webserver_port=38847
--webserver_interface=127.15.56.2
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.56.20:34445/kms with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:03:53 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403033, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:03:53.667194 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:53.646354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:49577 (local address 127.15.56.62:33479)
0810 23:03:53.647220 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:53.647244 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:53.647708 (+ 464us) server_negotiation.cc:407] Connection header received
0810 23:03:53.649179 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:53.649196 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:53.649622 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:53.649999 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:53.653403 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.655006 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.656891 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.657690 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.657850 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:53.660998 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:53.661006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:53.661028 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:53.661068 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:53.663145 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:53.663827 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:53.663833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:53.663839 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:53.663981 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:53.664380 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:53.664383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:53.664385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:53.664869 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:53.664987 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:53.665764 (+ 777us) server_negotiation.cc:299] Negotiation successful
0810 23:03:53.666090 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":111,"threads_started":1}
I20260810 23:03:53.668867 4629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:53.646509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33479 (local address 127.15.56.1:49577)
0810 23:03:53.647510 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:53.647552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:53.648529 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:53.652000 (+ 3471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:53.652010 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:53.652413 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:53.653146 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:53.653181 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.655236 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.655240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:53.656725 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:53.656738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.658166 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.658170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:53.658403 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:53.659096 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:53.659120 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:53.660710 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:53.663294 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:53.663301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:53.663317 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:53.663658 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:53.664090 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:53.664093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:53.664095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:53.664274 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:53.665057 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:53.665063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:53.665407 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 23:03:53.667563 (+ 2156us) client_negotiation.cc:240] Negotiation successful
0810 23:03:53.667927 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":118,"threads_started":1}
I20260810 23:03:53.671516 4626 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33479
I20260810 23:03:53.671800 4626 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:53.672322 4626 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
W20260810 23:03:53.674530 4626 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:33479 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:03:53.674649 4626 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:53.674830 4626 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
I20260810 23:03:53.675654 4626 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:53.675863 4626 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
W20260810 23:03:53.676589 4626 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:03:53.792989 4630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:53.793258 4630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:53.793329 4630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:53.797864 4630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:53.797945 4630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:53.798027 4630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:03:53.804140 4630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.56.20:34445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:39101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=38847
--tserver_master_addrs=127.15.56.62:33479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:53.805537 4630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:53.806951 4630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:53.820007 4638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:53.820390 4641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:53.820465 4639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:53.821116 4630 server_base.cc:1034] running on GCE node
I20260810 23:03:53.821554 4630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:53.822227 4630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:53.823415 4630 hybrid_clock.cc:648] HybridClock initialized: now 1786403033823345 us; error 85 us; skew 500 ppm
Aug 10 23:03:53 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403033, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:53.826448 4630 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:03:53.827378 4630 webserver.cc:460] Webserver started at http://127.15.56.2:38847/ using document root <none> and password file <none>
I20260810 23:03:53.828058 4630 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:53.828140 4630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:03:53 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403033, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:53.849879 4630 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:03:53.859309 4656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:53.861801 4630 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.000s
I20260810 23:03:53.861966 4630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "751ab888f49b42ebbba0dd6bae3e8842"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "7542e890ed2776e7dff086692360a4b5"
server_key_iv: "c7b41f95851e5f949b25f8d7982a8a12"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:53.862707 4630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:53.887883 4630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:53.892411 4630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:53.892647 4630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:53.893486 4630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:53.894624 4630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:53.894692 4630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:53.894754 4630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:53.894785 4630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:53.905957 4630 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:39101
I20260810 23:03:53.906090 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:39101 every 8 connection(s)
I20260810 23:03:53.907258 4630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:53.916760 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 4630
I20260810 23:03:53.917448 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.56.20:34445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:42217
--local_ip_for_outbound_sockets=127.15.56.3
--tserver_master_addrs=127.15.56.62:33479
--webserver_port=35121
--webserver_interface=127.15.56.3
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:46193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.56.20:34445/kms with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:03:53 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403033, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:03:53.934528 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:53.913973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:37741 (local address 127.15.56.62:33479)
0810 23:03:53.914116 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:53.914120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:53.915537 (+ 1417us) server_negotiation.cc:407] Connection header received
0810 23:03:53.917077 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:53.917081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:53.917151 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:53.917214 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:53.920012 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.920878 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.922804 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.923191 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.923278 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:53.929621 (+ 6343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:53.929626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:53.929629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:53.929668 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:53.931512 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:53.932313 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:53.932317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:53.932319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:53.932418 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:53.932856 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:53.932859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:53.932861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:53.933122 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:53.933204 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:53.934192 (+ 988us) server_negotiation.cc:299] Negotiation successful
0810 23:03:53.934340 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:03:53.936409 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:53.914557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33479 (local address 127.15.56.2:37741)
0810 23:03:53.915327 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:53.915394 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:53.916715 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:53.917901 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:53.917914 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:53.918858 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:53.919770 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:53.919798 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.921082 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.921086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:53.922637 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:53.922650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.923533 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.923537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:53.926382 (+ 2845us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:53.927245 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:53.927269 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:53.929327 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:53.931726 (+ 2399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:53.931733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:53.931758 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:53.932178 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:53.932526 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:53.932528 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:53.932530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:53.932731 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:53.933486 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:53.933493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:53.933888 (+ 395us) client_negotiation.cc:769] Sending connection context
0810 23:03:53.934804 (+ 916us) client_negotiation.cc:240] Negotiation successful
0810 23:03:53.935216 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":215,"threads_started":1}
I20260810 23:03:53.937845 4814 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33479
I20260810 23:03:53.938112 4814 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:53.938771 4814 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
W20260810 23:03:53.940661 4814 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:33479 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:03:53.940812 4814 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:53.940996 4814 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
I20260810 23:03:53.941839 4814 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:53.942011 4814 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
W20260810 23:03:53.942735 4814 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:03:54.071975 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:54.072350 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:54.072423 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:54.077579 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:54.077665 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:54.077754 4818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:03:54.085248 4818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:46193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.56.20:34445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:42217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=35121
--tserver_master_addrs=127.15.56.62:33479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:54.086410 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:54.087481 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:54.098994 4827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:54.099896 4830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:54.100419 4818 server_base.cc:1034] running on GCE node
W20260810 23:03:54.101016 4828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:54.101593 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:54.102547 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:54.103770 4818 hybrid_clock.cc:648] HybridClock initialized: now 1786403034103725 us; error 45 us; skew 500 ppm
Aug 10 23:03:54 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403034, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:54.111536 4818 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:03:54.112804 4818 webserver.cc:460] Webserver started at http://127.15.56.3:35121/ using document root <none> and password file <none>
I20260810 23:03:54.113565 4818 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:54.113683 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:03:54 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403034, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:54.137197 4818 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 23:03:54.140687 4842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:54.142511 4818 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:03:54.142771 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "f03ca5475323412fb8883fa2eaf55c52"
format_stamp: "Formatted at 2026-08-10 23:03:52 on dist-test-slave-6bbx"
server_key: "52164a26e253554adda4dc5b511d0054"
server_key_iv: "a3464136f6d5bbfd1c0fa975999962e5"
server_key_version: "kuduclusterkey@0"
I20260810 23:03:54.143615 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:54.160769 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:54.164551 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:54.164852 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:54.165707 4818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:54.166851 4818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:54.166913 4818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:54.166967 4818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:54.166990 4818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:54.181088 4818 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:42217
I20260810 23:03:54.181124 5007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:42217 every 8 connection(s)
I20260810 23:03:54.182755 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:54.191579 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 4818
Aug 10 23:03:54 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403034, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:03:54.198714 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:54.186765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:53109 (local address 127.15.56.62:33479)
0810 23:03:54.186906 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:54.186910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:54.186928 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:03:54.187789 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:54.187794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:54.187851 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:54.187943 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:54.189048 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.189885 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.191057 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.191401 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.191523 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:54.194571 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:54.194575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:54.194578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:54.194606 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:54.196300 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:54.196814 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:54.196818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:54.196820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:54.196897 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:54.197217 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:54.197220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:54.197222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:54.197390 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:54.197472 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:54.198138 (+ 666us) server_negotiation.cc:299] Negotiation successful
0810 23:03:54.198302 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:03:54.199788 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:54.185549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33479 (local address 127.15.56.3:53109)
0810 23:03:54.186160 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:54.186218 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:54.187487 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:54.188092 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:54.188099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:54.188374 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:54.188887 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:54.188902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.190017 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.190020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:54.190911 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:54.190920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.191581 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.191586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:54.191880 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:54.192649 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:54.192670 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:54.194388 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:54.196414 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:54.196420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:54.196438 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:54.196719 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:54.196977 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:54.196980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:54.196983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:54.197122 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:54.197527 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:54.197533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:54.197876 (+ 343us) client_negotiation.cc:769] Sending connection context
0810 23:03:54.198080 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 23:03:54.198413 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":165,"threads_started":1}
I20260810 23:03:54.201020 5009 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33479
I20260810 23:03:54.201280 5009 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:54.201387 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:54.192376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.15.56.62:33479)
0810 23:03:54.192640 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:54.192645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:54.193088 (+ 443us) server_negotiation.cc:407] Connection header received
0810 23:03:54.193200 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:54.193203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:54.193253 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:54.193333 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:54.194063 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.194731 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.195740 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.195958 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.196043 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:54.197500 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:54.197504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:54.197506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:54.197528 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:54.199137 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:54.199619 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:54.199623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:54.199624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:54.199737 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:54.200122 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:54.200125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:54.200138 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:54.200353 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:54.200525 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:54.200981 (+ 456us) server_negotiation.cc:299] Negotiation successful
0810 23:03:54.201179 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
I20260810 23:03:54.201869 5009 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
W20260810 23:03:54.203409 5009 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:33479 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:03:54.203536 5009 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:54.203744 5009 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
I20260810 23:03:54.204542 5009 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:54.204702 5009 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
W20260810 23:03:54.206014 5009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 23:03:54 dist-test-slave-6bbx krb5kdc[2781](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403034, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:54.676860 4626 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:54.677258 4626 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
I20260810 23:03:54.678745 4308 ts_manager.cc:194] Registered new tserver with Master: 6f19d1872ab042a3a9d4ead5843ec89d (127.15.56.1:41567)
I20260810 23:03:54.680647 4308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:49577
I20260810 23:03:54.942994 4814 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:54.943339 4814 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
I20260810 23:03:54.944355 4308 ts_manager.cc:194] Registered new tserver with Master: 751ab888f49b42ebbba0dd6bae3e8842 (127.15.56.2:39101)
I20260810 23:03:54.945125 4308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:37741
Aug 10 23:03:54 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403034, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:55.206243 5009 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:55.206589 5009 heartbeater.cc:511] Master 127.15.56.62:33479 requested a full tablet report, sending...
I20260810 23:03:55.207540 4308 ts_manager.cc:194] Registered new tserver with Master: f03ca5475323412fb8883fa2eaf55c52 (127.15.56.3:42217)
I20260810 23:03:55.208364 4308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:53109
I20260810 23:03:55.208832 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:03:55.225598 5042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.213700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.15.56.62:33479)
0810 23:03:55.213976 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:55.213981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:55.215980 (+ 1999us) server_negotiation.cc:407] Connection header received
0810 23:03:55.216196 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:55.216200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:55.216255 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:55.216318 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:55.217457 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.218327 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.219551 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.219958 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.220060 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.221888 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:55.221892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:55.221894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:55.221923 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:55.223742 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:55.224273 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:55.224278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:55.224280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:55.224376 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:55.224716 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:55.224719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:55.224721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:55.224935 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:55.225061 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:55.225297 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 23:03:55.225423 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":102,"threads_started":1}
Aug 10 23:03:55 dist-test-slave-6bbx krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403034, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for HTTP/127.15.56.20@KRBTEST.COM
I20260810 23:03:55.229583 4308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:55.241210 4308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:55.271118 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.258679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:41567 (local address 127.0.0.1:57466)
0810 23:03:55.260655 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:55.260667 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:55.260758 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:55.262296 (+ 1538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:55.262304 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:55.262331 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:55.262492 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.262503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.264269 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.264272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.265564 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.265575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.270503 (+ 4928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.270506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.270630 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.270677 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 23:03:55.270793 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 23:03:55.270928 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1907,"thread_start_us":92,"threads_started":1}
I20260810 23:03:55.271828 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.258859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.15.56.1:41567)
0810 23:03:55.259389 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:55.259397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:55.260857 (+ 1460us) server_negotiation.cc:407] Connection header received
0810 23:03:55.260951 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:55.260957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:55.261220 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:55.261372 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:55.262644 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.263874 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.265709 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.266555 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.266659 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.266841 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:55.271528 (+ 4687us) server_negotiation.cc:299] Negotiation successful
0810 23:03:55.271653 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":83,"threads_started":1}
I20260810 23:03:55.274647 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.258451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:39101 (local address 127.0.0.1:60656)
0810 23:03:55.260367 (+ 1916us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:55.260380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:55.260475 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:55.264029 (+ 3554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:55.264033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:55.264054 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:55.264159 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.264165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.269150 (+ 4985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.269154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.270372 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.270386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.274212 (+ 3826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.274216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.274316 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.274343 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:03:55.274413 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:03:55.274497 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1854,"thread_start_us":107,"threads_started":1}
I20260810 23:03:55.278103 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.258891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.15.56.2:39101)
0810 23:03:55.260224 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:55.260231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:55.261567 (+ 1336us) server_negotiation.cc:407] Connection header received
0810 23:03:55.261653 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:55.261658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:55.261865 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:55.262035 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:55.266928 (+ 4893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.268975 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.272980 (+ 4005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.273823 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.273936 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.274128 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:55.277735 (+ 3607us) server_negotiation.cc:299] Negotiation successful
0810 23:03:55.277950 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1221,"thread_start_us":131,"threads_started":1}
I20260810 23:03:55.280032 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.257220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:42217 (local address 127.0.0.1:40418)
0810 23:03:55.259705 (+ 2485us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:55.259770 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:55.259915 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:55.272689 (+ 12774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:55.272704 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:55.272725 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:55.272811 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.272817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.276006 (+ 3189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.276010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.277005 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.277015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.278925 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.278929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.279018 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.279041 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:03:55.279111 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:03:55.279201 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2262,"thread_start_us":133,"threads_started":1}
I20260810 23:03:55.280542 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.257330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.15.56.3:42217)
0810 23:03:55.268421 (+ 11091us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:55.268430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:55.268462 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:03:55.268560 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:55.268566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:55.272316 (+ 3750us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:55.272500 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:55.274750 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.275861 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.277747 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.278591 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.278703 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.278858 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:55.280296 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0810 23:03:55.280405 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10497,"thread_start_us":77,"threads_started":1}
I20260810 23:03:55.281145 4513 tablet_service.cc:1467] Processing CreateTablet for tablet c1cf1a644ec84ff9b865222203d2d7c9 (DEFAULT_TABLE table=test-table [id=acc7ffd77b0848079e88fb3d039ee47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:55.281414 4721 tablet_service.cc:1467] Processing CreateTablet for tablet c1cf1a644ec84ff9b865222203d2d7c9 (DEFAULT_TABLE table=test-table [id=acc7ffd77b0848079e88fb3d039ee47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:55.282671 4721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1cf1a644ec84ff9b865222203d2d7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:55.283969 4513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1cf1a644ec84ff9b865222203d2d7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:55.285596 4908 tablet_service.cc:1467] Processing CreateTablet for tablet c1cf1a644ec84ff9b865222203d2d7c9 (DEFAULT_TABLE table=test-table [id=acc7ffd77b0848079e88fb3d039ee47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:55.286728 4908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1cf1a644ec84ff9b865222203d2d7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:55.293633 5058 tablet_bootstrap.cc:492] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: Bootstrap starting.
I20260810 23:03:55.294073 5059 tablet_bootstrap.cc:492] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d: Bootstrap starting.
I20260810 23:03:55.296427 5059 tablet_bootstrap.cc:654] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:55.296427 5058 tablet_bootstrap.cc:654] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:55.297405 5058 log.cc:824] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:55.297441 5059 log.cc:824] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:55.300335 5058 tablet_bootstrap.cc:492] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: No bootstrap required, opened a new log
I20260810 23:03:55.300676 5058 ts_tablet_manager.cc:1397] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 23:03:55.301528 5066 tablet_bootstrap.cc:492] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52: Bootstrap starting.
I20260810 23:03:55.302353 5059 tablet_bootstrap.cc:492] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d: No bootstrap required, opened a new log
I20260810 23:03:55.302585 5059 ts_tablet_manager.cc:1397] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 23:03:55.304105 5066 tablet_bootstrap.cc:654] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:55.304641 5058 raft_consensus.cc:348] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03ca5475323412fb8883fa2eaf55c52" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42217 } } peers { permanent_uuid: "751ab888f49b42ebbba0dd6bae3e8842" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 39101 } } peers { permanent_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41567 } }
I20260810 23:03:55.304879 5058 raft_consensus.cc:374] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:55.304952 5058 raft_consensus.cc:729] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 751ab888f49b42ebbba0dd6bae3e8842, State: Initialized, Role: FOLLOWER
I20260810 23:03:55.305001 5066 log.cc:824] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:55.305477 5058 consensus_queue.cc:260] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03ca5475323412fb8883fa2eaf55c52" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42217 } } peers { permanent_uuid: "751ab888f49b42ebbba0dd6bae3e8842" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 39101 } } peers { permanent_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41567 } }
I20260810 23:03:55.306425 5059 raft_consensus.cc:348] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03ca5475323412fb8883fa2eaf55c52" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42217 } } peers { permanent_uuid: "751ab888f49b42ebbba0dd6bae3e8842" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 39101 } } peers { permanent_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41567 } }
I20260810 23:03:55.306730 5059 raft_consensus.cc:374] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:55.306746 5058 ts_tablet_manager.cc:1428] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:03:55.306833 5059 raft_consensus.cc:729] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f19d1872ab042a3a9d4ead5843ec89d, State: Initialized, Role: FOLLOWER
I20260810 23:03:55.307190 4814 heartbeater.cc:503] Master 127.15.56.62:33479 was elected leader, sending a full tablet report...
I20260810 23:03:55.307325 5059 consensus_queue.cc:260] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03ca5475323412fb8883fa2eaf55c52" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42217 } } peers { permanent_uuid: "751ab888f49b42ebbba0dd6bae3e8842" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 39101 } } peers { permanent_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41567 } }
I20260810 23:03:55.307962 5066 tablet_bootstrap.cc:492] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52: No bootstrap required, opened a new log
I20260810 23:03:55.308164 5066 ts_tablet_manager.cc:1397] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 23:03:55.308419 5059 ts_tablet_manager.cc:1428] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:03:55.308846 4626 heartbeater.cc:503] Master 127.15.56.62:33479 was elected leader, sending a full tablet report...
I20260810 23:03:55.312769 5066 raft_consensus.cc:348] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03ca5475323412fb8883fa2eaf55c52" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42217 } } peers { permanent_uuid: "751ab888f49b42ebbba0dd6bae3e8842" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 39101 } } peers { permanent_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41567 } }
I20260810 23:03:55.313130 5066 raft_consensus.cc:374] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:55.313241 5066 raft_consensus.cc:729] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03ca5475323412fb8883fa2eaf55c52, State: Initialized, Role: FOLLOWER
I20260810 23:03:55.313828 5066 consensus_queue.cc:260] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03ca5475323412fb8883fa2eaf55c52" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42217 } } peers { permanent_uuid: "751ab888f49b42ebbba0dd6bae3e8842" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 39101 } } peers { permanent_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41567 } }
I20260810 23:03:55.315053 5066 ts_tablet_manager.cc:1428] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 23:03:55.315454 5009 heartbeater.cc:503] Master 127.15.56.62:33479 was elected leader, sending a full tablet report...
I20260810 23:03:55.348516 5073 raft_consensus.cc:482] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:55.348840 5073 raft_consensus.cc:504] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03ca5475323412fb8883fa2eaf55c52" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42217 } } peers { permanent_uuid: "751ab888f49b42ebbba0dd6bae3e8842" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 39101 } } peers { permanent_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41567 } }
I20260810 23:03:55.351456 5073 leader_election.cc:288] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03ca5475323412fb8883fa2eaf55c52 (127.15.56.3:42217), 751ab888f49b42ebbba0dd6bae3e8842 (127.15.56.2:39101)
I20260810 23:03:55.356106 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.350869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:50903 (local address 127.15.56.2:39101)
0810 23:03:55.351006 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:55.351010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:55.351883 (+ 873us) server_negotiation.cc:407] Connection header received
0810 23:03:55.352075 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:55.352079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:55.352131 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:55.352178 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:55.352555 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.353336 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.354804 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.355373 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.355439 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.355494 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:55.355886 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 23:03:55.355968 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:03:55.356106 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.350963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:45615 (local address 127.15.56.3:42217)
0810 23:03:55.351072 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:55.351077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:55.351404 (+ 327us) server_negotiation.cc:407] Connection header received
0810 23:03:55.351711 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:55.351715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:55.351775 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:55.351831 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:55.352278 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.353036 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.354653 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.355260 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.355341 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.355414 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:55.355886 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 23:03:55.355968 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:03:55.356676 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.350435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:42217 (local address 127.15.56.1:45615)
0810 23:03:55.351305 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:55.351322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:55.351501 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:55.351975 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:55.351978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:55.351992 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:55.352138 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.352144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.353200 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.353203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.354512 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.354524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.355513 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.355517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.355634 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.355757 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 23:03:55.356473 (+ 716us) client_negotiation.cc:240] Negotiation successful
0810 23:03:55.356526 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":5,"thread_start_us":128,"threads_started":1}
I20260810 23:03:55.357609 4922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cf1a644ec84ff9b865222203d2d7c9" candidate_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03ca5475323412fb8883fa2eaf55c52" is_pre_election: true
I20260810 23:03:55.357882 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.350435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:39101 (local address 127.15.56.1:50903)
0810 23:03:55.351794 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:55.351807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:55.351956 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:55.352307 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:55.352311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:55.352326 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:55.352431 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.352437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.353498 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.353501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.354682 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.354691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.355570 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.355573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.355755 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.355774 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 23:03:55.356417 (+ 643us) client_negotiation.cc:240] Negotiation successful
0810 23:03:55.356496 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1294,"mutex_wait_us":105,"thread_start_us":102,"threads_started":1}
I20260810 23:03:55.358001 4922 raft_consensus.cc:2454] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f19d1872ab042a3a9d4ead5843ec89d in term 0.
I20260810 23:03:55.358529 4740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cf1a644ec84ff9b865222203d2d7c9" candidate_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751ab888f49b42ebbba0dd6bae3e8842" is_pre_election: true
I20260810 23:03:55.358636 4445 leader_election.cc:302] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f19d1872ab042a3a9d4ead5843ec89d, f03ca5475323412fb8883fa2eaf55c52; no voters:
I20260810 23:03:55.358853 4740 raft_consensus.cc:2454] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f19d1872ab042a3a9d4ead5843ec89d in term 0.
I20260810 23:03:55.358940 5073 raft_consensus.cc:2781] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:55.359074 5073 raft_consensus.cc:482] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:55.359186 5073 raft_consensus.cc:3037] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:55.360764 5073 raft_consensus.cc:504] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03ca5475323412fb8883fa2eaf55c52" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42217 } } peers { permanent_uuid: "751ab888f49b42ebbba0dd6bae3e8842" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 39101 } } peers { permanent_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41567 } }
I20260810 23:03:55.361599 4922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cf1a644ec84ff9b865222203d2d7c9" candidate_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03ca5475323412fb8883fa2eaf55c52"
I20260810 23:03:55.361733 4922 raft_consensus.cc:3037] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:55.363081 4922 raft_consensus.cc:2454] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f19d1872ab042a3a9d4ead5843ec89d in term 1.
I20260810 23:03:55.363518 4445 leader_election.cc:302] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f19d1872ab042a3a9d4ead5843ec89d, f03ca5475323412fb8883fa2eaf55c52; no voters:
I20260810 23:03:55.364353 4740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1cf1a644ec84ff9b865222203d2d7c9" candidate_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "751ab888f49b42ebbba0dd6bae3e8842"
I20260810 23:03:55.364498 4740 raft_consensus.cc:3037] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:55.365869 4740 raft_consensus.cc:2454] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f19d1872ab042a3a9d4ead5843ec89d in term 1.
I20260810 23:03:55.366324 5073 leader_election.cc:288] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [CANDIDATE]: Term 1 election: Requested vote from peers f03ca5475323412fb8883fa2eaf55c52 (127.15.56.3:42217), 751ab888f49b42ebbba0dd6bae3e8842 (127.15.56.2:39101)
I20260810 23:03:55.366551 5073 raft_consensus.cc:2781] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:55.366688 5073 raft_consensus.cc:686] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [term 1 LEADER]: Becoming Leader. State: Replica: 6f19d1872ab042a3a9d4ead5843ec89d, State: Running, Role: LEADER
I20260810 23:03:55.367113 5073 consensus_queue.cc:237] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03ca5475323412fb8883fa2eaf55c52" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42217 } } peers { permanent_uuid: "751ab888f49b42ebbba0dd6bae3e8842" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 39101 } } peers { permanent_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41567 } }
I20260810 23:03:55.370968 4307 catalog_manager.cc:5495] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f19d1872ab042a3a9d4ead5843ec89d (127.15.56.1). New cstate: current_term: 1 leader_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03ca5475323412fb8883fa2eaf55c52" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "751ab888f49b42ebbba0dd6bae3e8842" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 39101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f19d1872ab042a3a9d4ead5843ec89d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 41567 } health_report { overall_health: HEALTHY } } }
W20260810 23:03:55.397277 4627 tablet.cc:2369] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:03:55.400087 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.394868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.15.56.1:41567)
0810 23:03:55.394989 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:55.394993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:55.395008 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:03:55.395092 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:55.395095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:55.395153 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:55.395260 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:55.395555 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.396385 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.397706 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.398048 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.398170 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.398614 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:55.399231 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:55.399361 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:55.399846 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 23:03:55.399917 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:03:55.408046 4740 raft_consensus.cc:1261] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842 [term 1 FOLLOWER]: Refusing update from remote peer 6f19d1872ab042a3a9d4ead5843ec89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:55.408057 4922 raft_consensus.cc:1261] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52 [term 1 FOLLOWER]: Refusing update from remote peer 6f19d1872ab042a3a9d4ead5843ec89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:55.408828 5073 consensus_queue.cc:1035] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "751ab888f49b42ebbba0dd6bae3e8842" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 39101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:55.409332 5073 consensus_queue.cc:1035] T c1cf1a644ec84ff9b865222203d2d7c9 P 6f19d1872ab042a3a9d4ead5843ec89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03ca5475323412fb8883fa2eaf55c52" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:03:55.409647 4816 tablet.cc:2369] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:03:55.424876 5085 mvcc.cc:205] Tried to move back new op lower bound from 7317106833021931520 to 7317106832868282368. Current Snapshot: MvccSnapshot[applied={T|T < 7317106833021931520}]
I20260810 23:03:55.430150 5086 mvcc.cc:205] Tried to move back new op lower bound from 7317106833021931520 to 7317106832868282368. Current Snapshot: MvccSnapshot[applied={T|T < 7317106833021931520}]
I20260810 23:03:55.436188 4307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:55.436527 4307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:55.439697 4307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5afa7dd02923413b8162ea41990e263a: Sending DeleteTablet for 3 replicas of tablet c1cf1a644ec84ff9b865222203d2d7c9
I20260810 23:03:55.440588 4908 tablet_service.cc:1514] Processing DeleteTablet for tablet c1cf1a644ec84ff9b865222203d2d7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:55 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:40418
I20260810 23:03:55.441980 4721 tablet_service.cc:1514] Processing DeleteTablet for tablet c1cf1a644ec84ff9b865222203d2d7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:55 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:60656
I20260810 23:03:55.442525 4513 tablet_service.cc:1514] Processing DeleteTablet for tablet c1cf1a644ec84ff9b865222203d2d7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:55 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:57466
I20260810 23:03:55.442754 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 4368
I20260810 23:03:55.452900 5102 tablet_replica.cc:330] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: stopping tablet replica
I20260810 23:03:55.453392 5102 raft_consensus.cc:2229] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:55.454013 5102 raft_consensus.cc:2258] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:55.456022 5102 ts_tablet_manager.cc:1905] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:55.459051 5102 ts_tablet_manager.cc:1918] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:55.459244 5102 log.cc:1196] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/c1cf1a644ec84ff9b865222203d2d7c9
I20260810 23:03:55.459699 5102 ts_tablet_manager.cc:1939] T c1cf1a644ec84ff9b865222203d2d7c9 P 751ab888f49b42ebbba0dd6bae3e8842: Deleting consensus metadata
I20260810 23:03:55.461517 4291 catalog_manager.cc:4841] TS 751ab888f49b42ebbba0dd6bae3e8842 (127.15.56.2:39101): tablet c1cf1a644ec84ff9b865222203d2d7c9 (table test-table [id=acc7ffd77b0848079e88fb3d039ee47e]) successfully deleted
I20260810 23:03:55.467833 5101 tablet_replica.cc:330] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52: stopping tablet replica
I20260810 23:03:55.468413 5101 raft_consensus.cc:2229] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:55.469028 5101 raft_consensus.cc:2258] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:03:55.469681 4290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:41567 (error 108)
I20260810 23:03:55.472203 5101 ts_tablet_manager.cc:1905] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:55.473924 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 4630
I20260810 23:03:55.474223 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.473905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:41567 (local address 127.0.0.1:57478)
0810 23:03:55.474047 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:55.474127 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:03:55.476608 4290 catalog_manager.cc:4584] TS 6f19d1872ab042a3a9d4ead5843ec89d (127.15.56.1:41567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1cf1a644ec84ff9b865222203d2d7c9: Network error: Client connection negotiation failed: client connection to 127.15.56.1:41567: connect: Connection refused (error 111)
I20260810 23:03:55.485257 5101 ts_tablet_manager.cc:1918] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:55.485384 5101 log.cc:1196] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/c1cf1a644ec84ff9b865222203d2d7c9
I20260810 23:03:55.485680 5101 ts_tablet_manager.cc:1939] T c1cf1a644ec84ff9b865222203d2d7c9 P f03ca5475323412fb8883fa2eaf55c52: Deleting consensus metadata
I20260810 23:03:55.487314 4290 catalog_manager.cc:4841] TS f03ca5475323412fb8883fa2eaf55c52 (127.15.56.3:42217): tablet c1cf1a644ec84ff9b865222203d2d7c9 (table test-table [id=acc7ffd77b0848079e88fb3d039ee47e]) successfully deleted
I20260810 23:03:55.525688 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 4818
I20260810 23:03:55.549984 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.549574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:41567 (local address 127.0.0.1:57480)
0810 23:03:55.549806 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:55.549869 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260810 23:03:55.550998 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.550728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:41567 (local address 127.0.0.1:57486)
0810 23:03:55.550874 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:55.550922 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 23:03:55.556301 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 4268
I20260810 23:03:55.574697 15584 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:03:55.920960 15584 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:03:55.921087 15584 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:03:56.266572 15584 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:03:56.266 UTC [2824] LOG: received smart shutdown request
2026-08-10 23:03:56.271 UTC [2824] LOG: background worker "logical replication launcher" (PID 2835) exited with exit code 1
2026-08-10 23:03:56.271 UTC [2828] LOG: shutting down
2026-08-10 23:03:56.426 UTC [2824] LOG: database system is shut down
2026-08-10T23:03:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51222 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:56 dist-test-slave-6bbx krb5kdc[5117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:56 dist-test-slave-6bbx krb5kdc[5117](info): set up 2 sockets
Aug 10 23:03:56 dist-test-slave-6bbx krb5kdc[5117](info): commencing operation
krb5kdc: starting...
W20260810 23:03:57.692948 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:03:57 dist-test-slave-6bbx krb5kdc[5117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:03:57.827648 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:38551
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:38551,127.15.56.61:32947,127.15.56.60:40537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:57.948596 5136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:57.948896 5136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:57.948937 5136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:57.952927 5136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:57.952987 5136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:57.953006 5136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:57.953022 5136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:57.953040 5136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:57.958570 5136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:38551,127.15.56.61:32947,127.15.56.60:40537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:38551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:57.959862 5136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:57.961002 5136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:57.970189 5146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:57.970211 5142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:57.970189 5141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:57.970243 5136 server_base.cc:1034] running on GCE node
I20260810 23:03:57.971148 5136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:57.972440 5136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:57.973654 5136 hybrid_clock.cc:648] HybridClock initialized: now 1786403037973634 us; error 53 us; skew 500 ppm
Aug 10 23:03:57 dist-test-slave-6bbx krb5kdc[5117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403037, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:57.977097 5136 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:03:57.978134 5136 webserver.cc:460] Webserver started at http://127.15.56.62:36355/ using document root <none> and password file <none>
I20260810 23:03:57.978722 5136 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:57.978792 5136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:57.978981 5136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:57.981271 5136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "8dfc3174b10a416c9173372c4478388d"
format_stamp: "Formatted at 2026-08-10 23:03:57 on dist-test-slave-6bbx"
server_key: "4974b53f0dcce52a4010b59e7c19b193"
server_key_iv: "76f49d58a30c9b1efbc34b3ecc945909"
server_key_version: "encryptionkey@0"
I20260810 23:03:57.981868 5136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "8dfc3174b10a416c9173372c4478388d"
format_stamp: "Formatted at 2026-08-10 23:03:57 on dist-test-slave-6bbx"
server_key: "4974b53f0dcce52a4010b59e7c19b193"
server_key_iv: "76f49d58a30c9b1efbc34b3ecc945909"
server_key_version: "encryptionkey@0"
I20260810 23:03:57.985935 5136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:03:57.988511 5156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:57.989895 5136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:03:57.990031 5136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "8dfc3174b10a416c9173372c4478388d"
format_stamp: "Formatted at 2026-08-10 23:03:57 on dist-test-slave-6bbx"
server_key: "4974b53f0dcce52a4010b59e7c19b193"
server_key_iv: "76f49d58a30c9b1efbc34b3ecc945909"
server_key_version: "encryptionkey@0"
I20260810 23:03:57.990177 5136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:58.015338 5136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:58.018795 5136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:58.018987 5136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:58.026494 5136 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:38551
I20260810 23:03:58.026528 5234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:38551 every 8 connection(s)
I20260810 23:03:58.027549 5136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:58.030694 5235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:58.035215 5235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:03:58.036614 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 5136
I20260810 23:03:58.036753 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:03:58.037050 15584 external_mini_cluster.cc:1428] Setting key 635e9f1527e6cf006a3a9fb456339bb9
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403037, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.61@KRBTEST.COM" created.
I20260810 23:03:58.052318 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.038149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:38551 (local address 127.0.0.1:60000)
0810 23:03:58.038800 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.038868 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.039810 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.040506 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.040512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.040988 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.041333 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.041353 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.042563 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.042566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.043821 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.043833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.044484 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.044489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.044622 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.045246 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.045268 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.047066 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:58.049040 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.049051 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.049065 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.049306 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.049658 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.049660 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.049662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.049788 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.050501 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:58.050507 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:58.050676 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 23:03:58.050893 (+ 217us) client_negotiation.cc:240] Negotiation successful
0810 23:03:58.051312 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":43,"spinlock_wait_cycles":48384,"thread_start_us":171,"threads_started":1}
I20260810 23:03:58.052323 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.038149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.15.56.62:38551)
0810 23:03:58.038816 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.038837 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.039010 (+ 173us) server_negotiation.cc:407] Connection header received
0810 23:03:58.040118 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.040123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.040322 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.040494 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.041520 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.042427 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.043934 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.044320 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.044437 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.047263 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.047272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.047287 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.047314 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.048927 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.049427 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.049433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.049434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.049562 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.049895 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.049899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.049900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.050383 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.050497 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.051005 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.051229 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":171,"threads_started":1}
I20260810 23:03:58.054740 5235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:03:58.055531 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.055250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:32947 (local address 127.0.0.1:54098)
0810 23:03:58.055379 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.055466 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:32947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:03:58.055784 5159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.56.61:32947: connect: Connection refused (error 111)
I20260810 23:03:58.056674 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.056459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:32947 (local address 127.0.0.1:54110)
0810 23:03:58.056573 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.056619 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:32947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 23:03:58.056836 5235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.61:32947: Network error: Client connection negotiation failed: client connection to 127.15.56.61:32947: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab.
Entry for principal kudu/127.15.56.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab.
I20260810 23:03:58.067049 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.61:32947
--webserver_interface=127.15.56.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab
--principal=kudu/127.15.56.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:38551,127.15.56.61:32947,127.15.56.60:40537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:03:58.106117 5235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } attempt: 1
I20260810 23:03:58.107136 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.106750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:32947 (local address 127.0.0.1:54126)
0810 23:03:58.106963 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.107044 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:32947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 23:03:58.107918 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.107699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:32947 (local address 127.0.0.1:54140)
0810 23:03:58.107802 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.107850 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:32947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 23:03:58.108040 5235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.61:32947: Network error: Client connection negotiation failed: client connection to 127.15.56.61:32947: connect: Connection refused (error 111)
I20260810 23:03:58.176229 5235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } attempt: 2
I20260810 23:03:58.177183 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.176844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:32947 (local address 127.0.0.1:54144)
0810 23:03:58.177031 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.177098 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:32947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 23:03:58.177912 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.177690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:32947 (local address 127.0.0.1:54148)
0810 23:03:58.177799 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.177848 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:32947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:03:58.178030 5235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.61:32947: Network error: Client connection negotiation failed: client connection to 127.15.56.61:32947: connect: Connection refused (error 111)
W20260810 23:03:58.185050 5246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:58.185361 5246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:58.185424 5246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:58.189745 5246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:58.189815 5246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:58.189842 5246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:58.189865 5246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:58.189894 5246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:58.195503 5246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:38551,127.15.56.61:32947,127.15.56.60:40537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab
--principal=kudu/127.15.56.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.61:32947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.56.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:58.196702 5246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:58.197811 5246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:58.206676 5252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:58.206681 5259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:58.206786 5246 server_base.cc:1034] running on GCE node
W20260810 23:03:58.206676 5255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:58.207541 5246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:58.208721 5246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:58.209914 5246 hybrid_clock.cc:648] HybridClock initialized: now 1786403038209881 us; error 41 us; skew 500 ppm
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403038, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:58.213083 5246 init.cc:377] Logged in from keytab as kudu/127.15.56.61@KRBTEST.COM (short username kudu)
I20260810 23:03:58.214116 5246 webserver.cc:460] Webserver started at http://127.15.56.61:40161/ using document root <none> and password file <none>
I20260810 23:03:58.214702 5246 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:58.214774 5246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:58.214983 5246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:58.217271 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/data/instance:
uuid: "bd8f142c21e0456a8c9f7b483972c967"
format_stamp: "Formatted at 2026-08-10 23:03:58 on dist-test-slave-6bbx"
server_key: "8f03354e7fd9770acf2f8ed21ca777b3"
server_key_iv: "916ee13bc26e8c3bd52eb78828816925"
server_key_version: "encryptionkey@0"
I20260810 23:03:58.217876 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/wal/instance:
uuid: "bd8f142c21e0456a8c9f7b483972c967"
format_stamp: "Formatted at 2026-08-10 23:03:58 on dist-test-slave-6bbx"
server_key: "8f03354e7fd9770acf2f8ed21ca777b3"
server_key_iv: "916ee13bc26e8c3bd52eb78828816925"
server_key_version: "encryptionkey@0"
I20260810 23:03:58.221946 5246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:03:58.224967 5267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:58.226364 5246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:03:58.226508 5246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/wal
uuid: "bd8f142c21e0456a8c9f7b483972c967"
format_stamp: "Formatted at 2026-08-10 23:03:58 on dist-test-slave-6bbx"
server_key: "8f03354e7fd9770acf2f8ed21ca777b3"
server_key_iv: "916ee13bc26e8c3bd52eb78828816925"
server_key_version: "encryptionkey@0"
I20260810 23:03:58.226630 5246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:58.242911 5246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:58.246191 5246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:58.246393 5246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:58.253736 5246 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.61:32947
I20260810 23:03:58.253772 5343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.61:32947 every 8 connection(s)
I20260810 23:03:58.254779 5246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
I20260810 23:03:58.257869 5344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:58.261978 5344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:03:58.264864 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 5246
I20260810 23:03:58.265025 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-1/wal/instance
I20260810 23:03:58.265322 15584 external_mini_cluster.cc:1428] Setting key a5291f6455f35d20e505a4f8368d5d99
I20260810 23:03:58.269186 5235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } attempt: 3
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403038, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.60@KRBTEST.COM; defaulting to no policy
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403037, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.60@KRBTEST.COM" created.
I20260810 23:03:58.279011 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.269658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:32947 (local address 127.0.0.1:54162)
0810 23:03:58.269859 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.269877 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.269961 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.270626 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.270629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.270800 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.270907 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.270913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.272140 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.272144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.272680 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.272689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.273443 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.273447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.273565 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.274073 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.274088 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.275433 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:58.277274 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.277308 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.277310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.277540 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.277822 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.277825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.277827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.277923 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.278475 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:58.278478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:58.278583 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 23:03:58.278673 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:03:58.278836 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 23:03:58.279637 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.264165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.15.56.62:38551)
0810 23:03:58.264303 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.264307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.265297 (+ 990us) server_negotiation.cc:407] Connection header received
0810 23:03:58.266450 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.266454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.266523 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.266597 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.268300 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.268848 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.270506 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.270888 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.270972 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.274186 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.274190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.274192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.274218 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.276393 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.276992 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.276996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.276998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.277080 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.277537 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.277540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.277541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.277811 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.277899 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.279367 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.279499 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:03:58.280169 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.264543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:38551 (local address 127.0.0.1:60004)
0810 23:03:58.265110 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.265171 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.266174 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.266821 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.266828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.267314 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.268085 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.268106 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.269005 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.269009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.270349 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.270365 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.271189 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.271193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.271409 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.272111 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.272134 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.274043 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:58.276503 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.276512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.276525 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.276893 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.277201 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.277204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.277206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.277419 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.277907 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:58.277915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:58.278255 (+ 340us) client_negotiation.cc:769] Sending connection context
0810 23:03:58.279738 (+ 1483us) client_negotiation.cc:240] Negotiation successful
0810 23:03:58.279935 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"spinlock_wait_cycles":1792,"thread_start_us":146,"threads_started":1}
I20260810 23:03:58.280191 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.269736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.15.56.61:32947)
0810 23:03:58.270193 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.270203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.270223 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:03:58.270258 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.270263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.270431 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.271037 (+ 606us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.271160 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.272031 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.272795 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.273296 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.273412 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.275563 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.275572 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.275587 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.275614 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.277175 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.277638 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.277644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.277645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.277729 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.278028 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.278031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.278033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.278365 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.278491 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.278805 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.279133 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":163,"threads_started":1}
I20260810 23:03:58.281595 5344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:03:58.282773 5235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:03:58.283517 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.283220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60064)
0810 23:03:58.283392 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.283443 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260810 23:03:58.284364 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.284064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60070)
0810 23:03:58.284237 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.284295 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260810 23:03:58.284548 5235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:40537: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403038, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
I20260810 23:03:58.290160 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.282095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:32947 (local address 127.0.0.1:54174)
0810 23:03:58.282236 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.282251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.282318 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.282520 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.282523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.282700 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.282810 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.282816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.283507 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.283510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.284237 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.284248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.284774 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.284777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.284882 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.285390 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.285402 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.286913 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:58.288590 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.288594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.288596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.288847 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.289078 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.289081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.289083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.289190 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.289680 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:58.289683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:58.289779 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 23:03:58.289891 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 23:03:58.290021 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 23:03:58.290318 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.282187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.15.56.61:32947)
0810 23:03:58.282303 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.282307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.282321 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:03:58.282381 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.282384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.282429 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.282503 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.282911 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.283413 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.284338 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.284667 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.284756 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.287074 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.287077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.287080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.287103 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.288471 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.288934 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.288936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.288937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.289005 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.289354 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.289357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.289358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.289560 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.289649 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.289998 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.290117 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:03:58.291060 5344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:03:58.291904 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.291600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60084)
0810 23:03:58.291737 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.291843 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:03:58.292541 5274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
I20260810 23:03:58.293478 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.293264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60098)
0810 23:03:58.293373 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.293419 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 23:03:58.293670 5344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:40537: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab.
Entry for principal kudu/127.15.56.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab.
I20260810 23:03:58.296893 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.60:40537
--webserver_interface=127.15.56.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab
--principal=kudu/127.15.56.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:38551,127.15.56.61:32947,127.15.56.60:40537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:03:58.315773 5235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } attempt: 1
I20260810 23:03:58.316839 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.316410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60100)
0810 23:03:58.316627 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.316705 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260810 23:03:58.317643 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.317403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60116)
0810 23:03:58.317512 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.317568 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:03:58.317793 5235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:40537: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
I20260810 23:03:58.342948 5344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } attempt: 1
I20260810 23:03:58.343995 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.343536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60118)
0810 23:03:58.343786 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.343879 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 23:03:58.344843 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.344571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60128)
0810 23:03:58.344698 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.344767 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:03:58.344980 5344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:40537: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
I20260810 23:03:58.392938 5235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } attempt: 2
I20260810 23:03:58.393899 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.393508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60130)
0810 23:03:58.393704 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.393795 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 23:03:58.394723 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.394462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60138)
0810 23:03:58.394577 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.394636 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:03:58.394883 5235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:40537: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
I20260810 23:03:58.413163 5344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } attempt: 2
I20260810 23:03:58.414109 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.413767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60152)
0810 23:03:58.413948 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.414025 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 23:03:58.414908 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.414644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60160)
0810 23:03:58.414783 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.414839 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 23:03:58.415052 5344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:40537: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
W20260810 23:03:58.418699 5354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:58.419026 5354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:58.419072 5354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:58.423405 5354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:58.423494 5354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:58.423522 5354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:58.423543 5354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:58.423565 5354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:58.429359 5354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:38551,127.15.56.61:32947,127.15.56.60:40537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab
--principal=kudu/127.15.56.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.60:40537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.56.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:03:58.430640 5354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:58.431865 5354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:58.440718 5367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:58.440737 5365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:58.440728 5370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:58.441020 5354 server_base.cc:1034] running on GCE node
I20260810 23:03:58.441681 5354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:58.442888 5354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:58.444051 5354 hybrid_clock.cc:648] HybridClock initialized: now 1786403038444018 us; error 47 us; skew 500 ppm
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403038, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:58.446961 5354 init.cc:377] Logged in from keytab as kudu/127.15.56.60@KRBTEST.COM (short username kudu)
I20260810 23:03:58.447948 5354 webserver.cc:460] Webserver started at http://127.15.56.60:40843/ using document root <none> and password file <none>
I20260810 23:03:58.448498 5354 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:58.448556 5354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:58.448740 5354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:58.450893 5354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/data/instance:
uuid: "63194e74feca45f28ff4283de55513a9"
format_stamp: "Formatted at 2026-08-10 23:03:58 on dist-test-slave-6bbx"
server_key: "64b3d97632a2514bea9d1e8cc6ee658f"
server_key_iv: "63ae65056789d8ca1ef44ed9a5ee1af5"
server_key_version: "encryptionkey@0"
I20260810 23:03:58.451452 5354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/wal/instance:
uuid: "63194e74feca45f28ff4283de55513a9"
format_stamp: "Formatted at 2026-08-10 23:03:58 on dist-test-slave-6bbx"
server_key: "64b3d97632a2514bea9d1e8cc6ee658f"
server_key_iv: "63ae65056789d8ca1ef44ed9a5ee1af5"
server_key_version: "encryptionkey@0"
I20260810 23:03:58.455252 5354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:03:58.457773 5379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:58.459136 5354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:58.459256 5354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/wal
uuid: "63194e74feca45f28ff4283de55513a9"
format_stamp: "Formatted at 2026-08-10 23:03:58 on dist-test-slave-6bbx"
server_key: "64b3d97632a2514bea9d1e8cc6ee658f"
server_key_iv: "63ae65056789d8ca1ef44ed9a5ee1af5"
server_key_version: "encryptionkey@0"
I20260810 23:03:58.459373 5354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:58.487814 5354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:58.491341 5354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:58.491546 5354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:58.494053 5235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } attempt: 3
I20260810 23:03:58.494875 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.494531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60170)
0810 23:03:58.494698 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.494768 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:03:58.495770 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.495475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60184)
0810 23:03:58.495609 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.495693 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 23:03:58.495949 5235 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:40537: Network error: Client connection negotiation failed: client connection to 127.15.56.60:40537: connect: Connection refused (error 111)
I20260810 23:03:58.499280 5354 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.60:40537
I20260810 23:03:58.499286 5454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.60:40537 every 8 connection(s)
I20260810 23:03:58.500425 5354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
I20260810 23:03:58.503816 5455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:58.505051 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 5354
I20260810 23:03:58.505191 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402867637343-15584-0/minicluster-data/master-2/wal/instance
I20260810 23:03:58.505527 15584 external_mini_cluster.cc:1428] Setting key 4e99f35c18887b61c0b734a6ecc44fa5
I20260810 23:03:58.506227 5344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } attempt: 3
I20260810 23:03:58.509347 5455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403038, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403038, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:03:58.519564 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.506713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60212)
0810 23:03:58.507043 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.507061 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.507165 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.509214 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.509218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.509442 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.509579 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.509588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.511212 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.511216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.512080 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.512094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.513151 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.513156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.513247 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.513770 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.513785 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.515298 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:58.517514 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.517517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.517520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.517767 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.518107 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.518110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.518112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.518216 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.519044 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:58.519047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:58.519163 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 23:03:58.519253 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:03:58.519400 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260810 23:03:58.521394 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.511546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.15.56.62:38551)
0810 23:03:58.511707 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.511711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.512080 (+ 369us) server_negotiation.cc:407] Connection header received
0810 23:03:58.512308 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.512312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.512370 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.512451 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.513126 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.513766 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.514798 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.515017 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.515118 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.517712 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.517717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.517720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.517755 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.519328 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.519907 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.519911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.519912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.520010 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.520360 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.520363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.520365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.520584 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.520682 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.521122 (+ 440us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.521255 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:03:58.521471 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.507457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.15.56.60:40537)
0810 23:03:58.507892 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.507904 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.507922 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:03:58.508474 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.508479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.508889 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.509721 (+ 832us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.509859 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.511034 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.512229 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.513005 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.513168 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.515427 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.515432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.515442 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.515468 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.517374 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.517871 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.517877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.517884 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.518006 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.518338 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.518342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.518344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.518912 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.519845 (+ 933us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.520024 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.520288 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":418,"server-negotiator.queue_time_us":353,"spinlock_wait_cycles":24960,"thread_start_us":145,"threads_started":1}
I20260810 23:03:58.521471 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.507147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.15.56.60:40537)
0810 23:03:58.507649 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.507704 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.507758 (+ 54us) server_negotiation.cc:407] Connection header received
0810 23:03:58.508457 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.508476 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.508885 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.509223 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.510050 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.510985 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.512230 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.513005 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.513501 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.516153 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.516157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.516160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.516200 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.518909 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.519326 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.519329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.519331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.519416 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.519956 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.519960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.519962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.520162 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.520331 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.520671 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.521046 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":404,"server-negotiator.queue_time_us":336,"spinlock_wait_cycles":5888,"thread_start_us":162,"threads_started":1}
I20260810 23:03:58.521989 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.511479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:38551 (local address 127.0.0.1:60016)
0810 23:03:58.511959 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.512009 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.512169 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.512499 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.512503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.512878 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.512990 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.512996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.513906 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.513909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.514682 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.514693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.515181 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.515185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.515314 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.515880 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.515897 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.517546 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:58.519435 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.519445 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.519452 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.519775 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.520125 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.520128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.520130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.520240 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.520738 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:58.520744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:58.520865 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 23:03:58.521003 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 23:03:58.521191 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":108,"threads_started":1}
I20260810 23:03:58.523584 5455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:03:58.526840 5344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967: Bootstrap starting.
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:03:58.530002 5344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967: Neither blocks nor log segments found. Creating new log.
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403038, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
I20260810 23:03:58.531040 5344 log.cc:824] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:58.533911 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.524495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.15.56.62:38551)
0810 23:03:58.524630 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.524633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.524644 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:03:58.524676 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.524679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.524719 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.524787 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.525455 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.525968 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.526779 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.527030 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.527126 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.530409 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.530412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.530415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.530441 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.532140 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.532548 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.532552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.532554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.532634 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.532895 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.532899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.532900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.533163 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.533384 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.533510 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.533626 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:03:58.534261 5344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967: No bootstrap required, opened a new log
I20260810 23:03:58.534420 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.524271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54186 (local address 127.15.56.61:32947)
0810 23:03:58.524399 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.524404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.524704 (+ 300us) server_negotiation.cc:407] Connection header received
0810 23:03:58.524864 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.524867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.524914 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.524958 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.525593 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.526300 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.527328 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.527603 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.527732 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.531005 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.531009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.531012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.531065 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.532533 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.532998 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.533002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.533003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.533088 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.533391 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.533395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.533397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.533640 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.533713 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.534119 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.534252 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:03:58.534905 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.524106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:32947 (local address 127.0.0.1:54186)
0810 23:03:58.524633 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.524647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.524773 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.525103 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.525105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.525308 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.525447 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.525455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.526447 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.526450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.527186 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.527197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.527877 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.527880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.527990 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.528441 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.528457 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.530566 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:58.532643 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.532647 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.532649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.532894 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.533186 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.533189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.533190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.533280 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.533793 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:58.533796 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:58.533899 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 23:03:58.534577 (+ 678us) client_negotiation.cc:240] Negotiation successful
0810 23:03:58.534743 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452}
I20260810 23:03:58.535800 5455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:03:58.538640 5344 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } }
I20260810 23:03:58.538915 5344 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:58.538978 5344 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8f142c21e0456a8c9f7b483972c967, State: Initialized, Role: FOLLOWER
I20260810 23:03:58.539548 5344 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } }
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
I20260810 23:03:58.541137 5344 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403038, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
I20260810 23:03:58.545539 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.536498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.15.56.60:40537)
0810 23:03:58.536614 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.536618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.537343 (+ 725us) server_negotiation.cc:407] Connection header received
0810 23:03:58.537429 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.537432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.537476 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.537554 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.537967 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.538642 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.539492 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.539870 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.539963 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.541971 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.541975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.541978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.542001 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.543822 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.544249 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.544252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.544254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.544326 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.544593 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.544596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.544598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.544867 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.544934 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.545252 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.545379 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":35}
I20260810 23:03:58.545538 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.535209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.15.56.61:32947)
0810 23:03:58.535311 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.535314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.535395 (+ 81us) server_negotiation.cc:407] Connection header received
0810 23:03:58.535525 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.535529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.535570 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.535643 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.536349 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.537076 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.537994 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.538332 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.538431 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.541374 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.541379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.541382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.541411 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.543119 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.543638 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.543642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.543644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.543789 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.544113 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.544117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.544118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.544349 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.544521 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.545275 (+ 754us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.545379 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:03:58.546001 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.536290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60220)
0810 23:03:58.537266 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.537280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.537359 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.537578 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.537581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.537755 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.537851 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.537858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.538736 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.538739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.539384 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.539394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.539972 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.539975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.540070 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.540533 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.540548 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.541832 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:58.543922 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.543924 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.543926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.544144 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.544423 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.544426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.544427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.544508 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.544990 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:58.544993 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:58.545078 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 23:03:58.545718 (+ 640us) client_negotiation.cc:240] Negotiation successful
0810 23:03:58.545856 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":3}
W20260810 23:03:58.547250 5480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:03:58.547319 5480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:03:58.547503 5464 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } } }
I20260810 23:03:58.547622 5464 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:03:58.545301 5246 master_runner.cc:186] Error getting master registration for 127.15.56.62:38551: OK, instance_id { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" instance_seqno: 1786403038024314 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:03:58.548537 5246 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:03:58.549355 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:03:58.551584 5455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9: Bootstrap starting.
I20260810 23:03:58.554667 5455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:58.555872 5455 log.cc:824] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:58.559320 5455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9: No bootstrap required, opened a new log
I20260810 23:03:58.560348 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.551370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.15.56.62:38551)
0810 23:03:58.551506 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.551510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.552328 (+ 818us) server_negotiation.cc:407] Connection header received
0810 23:03:58.552440 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.552443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.552494 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.552561 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.553275 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.553784 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.554592 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.554821 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.554914 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.556531 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.556535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.556538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.556572 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.558240 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.558750 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.558754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.558755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.558874 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.559211 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.559214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.559215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.559467 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.559576 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.560023 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.560174 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:03:58.566296 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.551334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.15.56.60:40537)
0810 23:03:58.551449 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.551454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.556696 (+ 5242us) server_negotiation.cc:407] Connection header received
0810 23:03:58.556898 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.556901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.556950 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.557002 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.557867 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.558653 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.559970 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.560322 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.560405 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.562560 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.562564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.562567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.562597 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.564323 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.564807 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.564811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.564813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.564899 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.565232 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.565236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.565238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.565496 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.565652 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.566004 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.566133 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:03:58.566411 5455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } }
I20260810 23:03:58.566958 5455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:58.567022 5455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63194e74feca45f28ff4283de55513a9, State: Initialized, Role: FOLLOWER
I20260810 23:03:58.568522 5455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } }
I20260810 23:03:58.569154 5491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } } }
I20260810 23:03:58.569372 5491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:03:58.569948 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.551868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.15.56.61:32947)
0810 23:03:58.551983 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.551987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.560971 (+ 8984us) server_negotiation.cc:407] Connection header received
0810 23:03:58.561172 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.561175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.561233 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.561286 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.562169 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.562952 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.564292 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.564633 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.564715 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.566284 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.566288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.566291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.566319 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.568021 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.568497 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.568500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.568502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.568597 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.568939 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.568942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.568943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.569168 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.569290 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.569657 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.569795 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:03:58.570389 5455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 23:03:58.576262 5503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:03:58.576362 5503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:03:58.574007 5354 master_runner.cc:186] Error getting master registration for 127.15.56.62:38551: OK, instance_id { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" instance_seqno: 1786403038024314 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:03:58.576849 5354 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:03:58.660157 5235 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } attempt: 4
Aug 10 23:03:58 dist-test-slave-6bbx krb5kdc[5117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403037, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
I20260810 23:03:58.668919 5464 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:58.669124 5464 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } }
I20260810 23:03:58.670032 5238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.660639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:40537 (local address 127.0.0.1:60228)
0810 23:03:58.660835 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.660850 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.660931 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.661190 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.661193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.661427 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.661547 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.661555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.662595 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.662599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.663487 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.663500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.664091 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.664095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.664197 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.664780 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.664796 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.666430 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:58.668407 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.668410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.668412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.668664 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.668974 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:58.668977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:58.668978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:58.669066 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:58.669530 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:58.669533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:58.669634 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 23:03:58.669727 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:03:58.669868 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260810 23:03:58.670243 5464 leader_election.cc:288] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dfc3174b10a416c9173372c4478388d (127.15.56.62:38551), 63194e74feca45f28ff4283de55513a9 (127.15.56.60:40537)
I20260810 23:03:58.670636 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.660764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.15.56.60:40537)
0810 23:03:58.660890 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.660895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.660911 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:03:58.661017 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.661020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.661066 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.661162 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.661664 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.662487 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.663596 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.663960 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.664069 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.666594 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:58.666598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:58.666602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:58.666630 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:58.668287 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.668783 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.668787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.668789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.668876 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:58.669162 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:58.669165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:58.669166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:58.669396 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:58.669484 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:58.670319 (+ 835us) server_negotiation.cc:299] Negotiation successful
0810 23:03:58.670442 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:03:58.674124 5235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d: Bootstrap starting.
I20260810 23:03:58.677156 5235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:58.678103 5235 log.cc:824] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:58.679148 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd8f142c21e0456a8c9f7b483972c967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63194e74feca45f28ff4283de55513a9" is_pre_election: true
I20260810 23:03:58.679508 5418 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd8f142c21e0456a8c9f7b483972c967 in term 0.
I20260810 23:03:58.680133 5274 leader_election.cc:302] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63194e74feca45f28ff4283de55513a9, bd8f142c21e0456a8c9f7b483972c967; no voters:
I20260810 23:03:58.680583 5464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:58.680691 5464 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:58.680739 5464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:58.682039 5464 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } }
I20260810 23:03:58.682631 5464 leader_election.cc:288] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [CANDIDATE]: Term 1 election: Requested vote from peers 8dfc3174b10a416c9173372c4478388d (127.15.56.62:38551), 63194e74feca45f28ff4283de55513a9 (127.15.56.60:40537)
I20260810 23:03:58.683624 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd8f142c21e0456a8c9f7b483972c967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63194e74feca45f28ff4283de55513a9"
I20260810 23:03:58.683777 5418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:58.684978 5418 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd8f142c21e0456a8c9f7b483972c967 in term 1.
I20260810 23:03:58.685252 5274 leader_election.cc:302] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63194e74feca45f28ff4283de55513a9, bd8f142c21e0456a8c9f7b483972c967; no voters:
I20260810 23:03:58.685426 5464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:58.685660 5464 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [term 1 LEADER]: Becoming Leader. State: Replica: bd8f142c21e0456a8c9f7b483972c967, State: Running, Role: LEADER
I20260810 23:03:58.686003 5464 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } }
I20260810 23:03:58.687950 5464 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd8f142c21e0456a8c9f7b483972c967. Latest consensus state: current_term: 1 leader_uuid: "bd8f142c21e0456a8c9f7b483972c967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } } }
I20260810 23:03:58.688071 5464 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:58.691313 5508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:58.692127 5235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d: No bootstrap required, opened a new log
I20260810 23:03:58.693945 5508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:58.696684 5235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } }
I20260810 23:03:58.697139 5235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:58.697257 5235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfc3174b10a416c9173372c4478388d, State: Initialized, Role: FOLLOWER
I20260810 23:03:58.697856 5235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } }
I20260810 23:03:58.699388 5235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:58.699486 5190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd8f142c21e0456a8c9f7b483972c967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfc3174b10a416c9173372c4478388d"
I20260810 23:03:58.699788 5190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:58.701400 5190 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd8f142c21e0456a8c9f7b483972c967 in term 1.
I20260810 23:03:58.701686 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd8f142c21e0456a8c9f7b483972c967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dfc3174b10a416c9173372c4478388d" is_pre_election: true
I20260810 23:03:58.701825 5193 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate bd8f142c21e0456a8c9f7b483972c967 in term 1. Re-sending same reply.
I20260810 23:03:58.706712 5193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [term 1 FOLLOWER]: Refusing update from remote peer bd8f142c21e0456a8c9f7b483972c967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:58.706701 5418 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [term 1 FOLLOWER]: Refusing update from remote peer bd8f142c21e0456a8c9f7b483972c967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:58.707453 5506 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:58.707645 5464 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:58.712049 5531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd8f142c21e0456a8c9f7b483972c967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } } }
I20260810 23:03:58.712210 5531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:03:58.712311 5531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd8f142c21e0456a8c9f7b483972c967. Latest consensus state: current_term: 1 leader_uuid: "bd8f142c21e0456a8c9f7b483972c967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } } }
I20260810 23:03:58.712379 5531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:03:58.714321 5491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd8f142c21e0456a8c9f7b483972c967. Latest consensus state: current_term: 1 leader_uuid: "bd8f142c21e0456a8c9f7b483972c967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } } }
I20260810 23:03:58.714457 5491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:03:58.716828 5464 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd8f142c21e0456a8c9f7b483972c967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } } }
I20260810 23:03:58.716957 5464 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:58.717088 5532 mvcc.cc:205] Tried to move back new op lower bound from 7317106846530502656 to 7317106846461198336. Current Snapshot: MvccSnapshot[applied={T|T < 7317106846530502656}]
I20260810 23:03:58.717232 5531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd8f142c21e0456a8c9f7b483972c967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } } }
I20260810 23:03:58.717324 5531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:03:58.718930 5529 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:03:58.719024 5529 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:03:58.719621 5508 catalog_manager.cc:1347] Generated new cluster ID: c70ef970be1c474380e276c1ef0714f7
I20260810 23:03:58.719741 5508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:58.728664 5464 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd8f142c21e0456a8c9f7b483972c967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } } }
I20260810 23:03:58.728945 5464 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:58.730571 5491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd8f142c21e0456a8c9f7b483972c967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfc3174b10a416c9173372c4478388d" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38551 } } peers { permanent_uuid: "bd8f142c21e0456a8c9f7b483972c967" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 32947 } } peers { permanent_uuid: "63194e74feca45f28ff4283de55513a9" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 40537 } } }
I20260810 23:03:58.730721 5491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:03:58.731359 5534 mvcc.cc:205] Tried to move back new op lower bound from 7317106846530502656 to 7317106846461198336. Current Snapshot: MvccSnapshot[applied={T|T < 7317106846530502656}]
I20260810 23:03:58.740099 5508 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:58.741468 5508 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:58.755304 5508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd8f142c21e0456a8c9f7b483972c967: Generated new TSK 0
I20260810 23:03:58.756395 5508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:03:59.577721 5503 catalog_manager.cc:1259] Loaded cluster ID: c70ef970be1c474380e276c1ef0714f7
I20260810 23:03:59.577817 5503 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9: loading cluster ID for follower catalog manager: success
I20260810 23:03:59.580269 5503 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9: acquiring CA information for follower catalog manager: success
I20260810 23:03:59.581576 5503 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 63194e74feca45f28ff4283de55513a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:03:59.720513 5529 catalog_manager.cc:1259] Loaded cluster ID: c70ef970be1c474380e276c1ef0714f7
I20260810 23:03:59.720607 5529 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d: loading cluster ID for follower catalog manager: success
I20260810 23:03:59.722390 5529 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d: acquiring CA information for follower catalog manager: success
I20260810 23:03:59.723636 5529 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8dfc3174b10a416c9173372c4478388d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:03:59.829775 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 5136
I20260810 23:03:59.843369 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 5246
I20260810 23:03:59.856055 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 5354
2026-08-10T23:03:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3331 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:59 dist-test-slave-6bbx krb5kdc[5554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:59 dist-test-slave-6bbx krb5kdc[5554](info): set up 2 sockets
Aug 10 23:03:59 dist-test-slave-6bbx krb5kdc[5554](info): commencing operation
krb5kdc: starting...
W20260810 23:04:01.025175 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:04:01 dist-test-slave-6bbx krb5kdc[5554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:04:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:04:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:04:01.158460 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:35043
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:35043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:01.273878 5572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:01.274171 5572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:01.274228 5572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:01.278246 5572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:04:01.278324 5572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:01.278350 5572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:01.278371 5572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:04:01.278393 5572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:04:01.283761 5572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:35043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:35043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:01.284924 5572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:01.286041 5572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:04:01.295176 5572 server_base.cc:1034] running on GCE node
W20260810 23:04:01.295126 5578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:01.295248 5577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:01.295133 5580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:01.295990 5572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:01.297164 5572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:01.298331 5572 hybrid_clock.cc:648] HybridClock initialized: now 1786403041298316 us; error 28 us; skew 500 ppm
Aug 10 23:04:01 dist-test-slave-6bbx krb5kdc[5554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403041, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:01.301100 5572 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:04:01.302017 5572 webserver.cc:460] Webserver started at http://127.15.56.62:42395/ using document root <none> and password file <none>
I20260810 23:04:01.302587 5572 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:01.302656 5572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:01.302882 5572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:01.305053 5572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "e53d51ab13d641758adc1f7fbb3c6079"
format_stamp: "Formatted at 2026-08-10 23:04:01 on dist-test-slave-6bbx"
server_key: "596918740b7739be7e907f1de3c521ba"
server_key_iv: "c6da0b73ad04ef0434a9dd6551d2a830"
server_key_version: "encryptionkey@0"
I20260810 23:04:01.305603 5572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "e53d51ab13d641758adc1f7fbb3c6079"
format_stamp: "Formatted at 2026-08-10 23:04:01 on dist-test-slave-6bbx"
server_key: "596918740b7739be7e907f1de3c521ba"
server_key_iv: "c6da0b73ad04ef0434a9dd6551d2a830"
server_key_version: "encryptionkey@0"
I20260810 23:04:01.309433 5572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 23:04:01.311946 5592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:01.313218 5572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:04:01.313349 5572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "e53d51ab13d641758adc1f7fbb3c6079"
format_stamp: "Formatted at 2026-08-10 23:04:01 on dist-test-slave-6bbx"
server_key: "596918740b7739be7e907f1de3c521ba"
server_key_iv: "c6da0b73ad04ef0434a9dd6551d2a830"
server_key_version: "encryptionkey@0"
I20260810 23:04:01.313450 5572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:01.328897 5572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:01.331873 5572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:01.332073 5572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:01.339119 5572 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:35043
I20260810 23:04:01.339149 5678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:35043 every 8 connection(s)
I20260810 23:04:01.340210 5572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:04:01.343482 5679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:01.345715 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 5572
I20260810 23:04:01.345846 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:04:01.346132 15584 external_mini_cluster.cc:1428] Setting key 7343325e215d139454ba5537c9ef0b90
I20260810 23:04:01.350986 5679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079: Bootstrap starting.
I20260810 23:04:01.353989 5679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:01.355000 5679 log.cc:824] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:04:01 dist-test-slave-6bbx krb5kdc[5554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:01.357614 5679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079: No bootstrap required, opened a new log
I20260810 23:04:01.362783 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:01.347702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.15.56.62:35043)
0810 23:04:01.348190 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:01.348207 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:01.348248 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:04:01.348943 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:01.348963 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:01.349361 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:01.349675 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:01.350682 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.351865 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.353201 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.353739 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.353876 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:01.356684 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:01.356691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:01.356712 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:01.356753 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:01.358910 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:01.359459 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:01.359466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:01.359474 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:01.359641 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:01.360006 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:01.360010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:01.360012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:01.360669 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:01.360884 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:01.361246 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 23:04:01.361608 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":138,"threads_started":1}
I20260810 23:04:01.362625 5679 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d51ab13d641758adc1f7fbb3c6079" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35043 } }
I20260810 23:04:01.362936 5679 raft_consensus.cc:374] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:01.362985 5679 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53d51ab13d641758adc1f7fbb3c6079, State: Initialized, Role: FOLLOWER
I20260810 23:04:01.363598 5679 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d51ab13d641758adc1f7fbb3c6079" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35043 } }
I20260810 23:04:01.363830 5679 raft_consensus.cc:388] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:04:01.363902 5679 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:04:01.364001 5679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:01.365121 5679 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d51ab13d641758adc1f7fbb3c6079" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35043 } }
I20260810 23:04:01.365510 5679 leader_election.cc:302] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53d51ab13d641758adc1f7fbb3c6079; no voters:
I20260810 23:04:01.365896 5679 leader_election.cc:288] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:04:01.366019 5686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:01.366397 5686 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [term 1 LEADER]: Becoming Leader. State: Replica: e53d51ab13d641758adc1f7fbb3c6079, State: Running, Role: LEADER
I20260810 23:04:01.366683 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:04:01.366781 5686 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d51ab13d641758adc1f7fbb3c6079" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35043 } }
I20260810 23:04:01.367447 5679 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:04:01.368634 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53d51ab13d641758adc1f7fbb3c6079. Latest consensus state: current_term: 1 leader_uuid: "e53d51ab13d641758adc1f7fbb3c6079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d51ab13d641758adc1f7fbb3c6079" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35043 } } }
I20260810 23:04:01.368783 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:01.369148 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e53d51ab13d641758adc1f7fbb3c6079" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d51ab13d641758adc1f7fbb3c6079" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35043 } } }
I20260810 23:04:01.369256 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079 [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:01.369261 5696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:04:01.372769 5696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:04:01.378845 5696 catalog_manager.cc:1347] Generated new cluster ID: 0c6fea127faa4342bff41a5ba93585e1
I20260810 23:04:01.378929 5696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:04:01.384670 5696 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:04:01.386029 5696 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:04:01.392293 5696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e53d51ab13d641758adc1f7fbb3c6079: Generated new TSK 0
I20260810 23:04:01.393165 5696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:04:01.423736 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 5572
2026-08-10T23:04:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1570 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:04:01 dist-test-slave-6bbx krb5kdc[5717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:04:01 dist-test-slave-6bbx krb5kdc[5717](info): set up 2 sockets
Aug 10 23:04:01 dist-test-slave-6bbx krb5kdc[5717](info): commencing operation
krb5kdc: starting...
W20260810 23:04:02.594488 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:04:02 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:04:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:04:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:04:02.724104 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44547
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:44547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:02.844288 5738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:02.844607 5738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:02.844667 5738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:02.848810 5738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:04:02.848887 5738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:02.848913 5738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:02.848933 5738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:04:02.848955 5738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:04:02.854542 5738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:44547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:02.855724 5738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:02.856818 5738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:02.865839 5747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:02.865829 5750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:02.865898 5738 server_base.cc:1034] running on GCE node
W20260810 23:04:02.865825 5752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:02.866674 5738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:02.867959 5738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:02.869130 5738 hybrid_clock.cc:648] HybridClock initialized: now 1786403042869109 us; error 33 us; skew 500 ppm
Aug 10 23:04:02 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403042, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:02.872023 5738 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:04:02.872961 5738 webserver.cc:460] Webserver started at http://127.15.56.62:32949/ using document root <none> and password file <none>
I20260810 23:04:02.873528 5738 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:02.873590 5738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:02.873803 5738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:02.875998 5738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "c8ffc07682a24ae2b5da1d1b1db3003e"
format_stamp: "Formatted at 2026-08-10 23:04:02 on dist-test-slave-6bbx"
server_key: "7124602168ea99fce4d2af84347c31a4"
server_key_iv: "3bbfc056c10a7a80d7f2467920425f26"
server_key_version: "encryptionkey@0"
I20260810 23:04:02.876574 5738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "c8ffc07682a24ae2b5da1d1b1db3003e"
format_stamp: "Formatted at 2026-08-10 23:04:02 on dist-test-slave-6bbx"
server_key: "7124602168ea99fce4d2af84347c31a4"
server_key_iv: "3bbfc056c10a7a80d7f2467920425f26"
server_key_version: "encryptionkey@0"
I20260810 23:04:02.880537 5738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:04:02.883142 5764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:02.884477 5738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:04:02.884608 5738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "c8ffc07682a24ae2b5da1d1b1db3003e"
format_stamp: "Formatted at 2026-08-10 23:04:02 on dist-test-slave-6bbx"
server_key: "7124602168ea99fce4d2af84347c31a4"
server_key_iv: "3bbfc056c10a7a80d7f2467920425f26"
server_key_version: "encryptionkey@0"
I20260810 23:04:02.884727 5738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:02.917629 5738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:02.920943 5738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:02.921140 5738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:02.928314 5738 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:44547
I20260810 23:04:02.928604 5829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:44547 every 8 connection(s)
I20260810 23:04:02.929373 5738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:04:02.932333 5830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:02.932498 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 5738
I20260810 23:04:02.932606 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:04:02.932869 15584 external_mini_cluster.cc:1428] Setting key 5b0e4a0b42c0b3d6cef885ae1e561b8e
I20260810 23:04:02.939909 5830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Bootstrap starting.
I20260810 23:04:02.942543 5830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Neither blocks nor log segments found. Creating new log.
Aug 10 23:04:02 dist-test-slave-6bbx krb5kdc[5717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:02.943444 5830 log.cc:824] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:02.946075 5830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: No bootstrap required, opened a new log
I20260810 23:04:02.948872 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:02.934389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.15.56.62:44547)
0810 23:04:02.934850 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:02.934870 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:02.934916 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:04:02.935690 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:02.935719 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:02.936182 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:02.936586 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:02.937316 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.938851 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.939825 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.940523 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.940686 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:02.943166 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:02.943173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:02.943192 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:02.943229 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:02.945302 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:02.945760 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:02.945767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:02.945774 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:02.945927 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:02.946272 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:02.946275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:02.946277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:02.946863 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:02.947095 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:02.947367 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 23:04:02.947750 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":119,"threads_started":1}
I20260810 23:04:02.950802 5830 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:02.951115 5830 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:02.951187 5830 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ffc07682a24ae2b5da1d1b1db3003e, State: Initialized, Role: FOLLOWER
I20260810 23:04:02.951714 5830 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:02.951930 5830 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:04:02.952001 5830 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:04:02.952105 5830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:02.953186 5830 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:02.953543 5830 leader_election.cc:302] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8ffc07682a24ae2b5da1d1b1db3003e; no voters:
I20260810 23:04:02.953913 5830 leader_election.cc:288] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:04:02.954051 5839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:02.954336 5839 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 1 LEADER]: Becoming Leader. State: Replica: c8ffc07682a24ae2b5da1d1b1db3003e, State: Running, Role: LEADER
I20260810 23:04:02.954712 5839 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:02.955348 5830 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:04:02.956504 5843 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8ffc07682a24ae2b5da1d1b1db3003e. Latest consensus state: current_term: 1 leader_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } } }
I20260810 23:04:02.956532 5842 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } } }
I20260810 23:04:02.956638 5842 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:02.956638 5843 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:02.957072 5855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:04:02.960628 5855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:04:02.967358 5855 catalog_manager.cc:1347] Generated new cluster ID: ac64f1c9168e437c9fe7a20143006f40
I20260810 23:04:02.967455 5855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:04:02.976209 5855 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:04:02.977221 5855 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:04:02.991189 5855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Generated new TSK 0
I20260810 23:04:02.992112 5855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:04:03.030890 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44547
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:03.152661 5874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:03.152972 5874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:03.153034 5874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:03.157307 5874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:03.157384 5874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:03.157464 5874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:04:03.162957 5874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:03.164173 5874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:03.165282 5874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:03.175518 5882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:03.175557 5874 server_base.cc:1034] running on GCE node
W20260810 23:04:03.175537 5879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:03.175518 5880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:03.176184 5874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:03.176846 5874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:03.178004 5874 hybrid_clock.cc:648] HybridClock initialized: now 1786403043177978 us; error 41 us; skew 500 ppm
Aug 10 23:04:03 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:03.180811 5874 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:04:03.181730 5874 webserver.cc:460] Webserver started at http://127.15.56.1:41723/ using document root <none> and password file <none>
I20260810 23:04:03.182327 5874 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:03.182397 5874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:03.182621 5874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:03.184866 5874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "71725911b2a44f989f68a583ba8a0004"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "06bdc983af5ce1a6ff10ab211bdfeccd"
server_key_iv: "27b6798194683e42142149919290c741"
server_key_version: "encryptionkey@0"
I20260810 23:04:03.185451 5874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "71725911b2a44f989f68a583ba8a0004"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "06bdc983af5ce1a6ff10ab211bdfeccd"
server_key_iv: "27b6798194683e42142149919290c741"
server_key_version: "encryptionkey@0"
I20260810 23:04:03.189483 5874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 23:04:03.192051 5891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:03.193334 5874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:04:03.193454 5874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "71725911b2a44f989f68a583ba8a0004"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "06bdc983af5ce1a6ff10ab211bdfeccd"
server_key_iv: "27b6798194683e42142149919290c741"
server_key_version: "encryptionkey@0"
I20260810 23:04:03.193557 5874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:03.227957 5874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:03.231024 5874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:03.231217 5874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:03.231889 5874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:03.232771 5874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:03.232834 5874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:03.232892 5874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:03.232918 5874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:03.241516 5874 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:40503
I20260810 23:04:03.241568 6059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:40503 every 8 connection(s)
I20260810 23:04:03.242458 5874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:04:03.249542 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 5874
I20260810 23:04:03.249675 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:04:03.249936 15584 external_mini_cluster.cc:1428] Setting key 2c97e3a98576cb8cd53a810b31f5c6e7
Aug 10 23:04:03 dist-test-slave-6bbx krb5kdc[5717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:03.256932 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.244421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:43423 (local address 127.15.56.62:44547)
0810 23:04:03.244590 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.244593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.245503 (+ 910us) server_negotiation.cc:407] Connection header received
0810 23:04:03.246952 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.246956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.247019 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.247093 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:03.248429 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.248984 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.250236 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.250543 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.250639 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.253221 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:03.253226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:03.253228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:03.253256 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:03.254642 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:03.255234 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:03.255237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:03.255239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:03.255328 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:03.255698 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:03.255702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:03.255704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:03.255913 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:03.256006 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:03.256609 (+ 603us) server_negotiation.cc:299] Negotiation successful
0810 23:04:03.256758 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:04:03.257851 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.244796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:43423)
0810 23:04:03.245327 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.245375 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:03.246721 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:03.247302 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:03.247310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:03.247752 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:03.248270 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.248285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.249115 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.249117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.250109 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.250120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.250700 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.250704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.250875 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.251482 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:03.251500 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:03.253048 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:03.254765 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:03.254771 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:03.254788 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:03.255131 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:03.255415 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:03.255418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:03.255420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:03.255572 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:03.256023 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:03.256030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:03.256364 (+ 334us) client_negotiation.cc:769] Sending connection context
0810 23:04:03.256549 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 23:04:03.256903 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":116,"threads_started":1}
I20260810 23:04:03.259377 6061 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44547
I20260810 23:04:03.259634 6061 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:04:03.260213 6061 heartbeater.cc:511] Master 127.15.56.62:44547 requested a full tablet report, sending...
I20260810 23:04:03.261823 5784 ts_manager.cc:194] Registered new tserver with Master: 71725911b2a44f989f68a583ba8a0004 (127.15.56.1:40503)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
I20260810 23:04:03.263291 5784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:43423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:04:03.279351 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44547
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:03.398731 6068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:03.399040 6068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:03.399085 6068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:03.403231 6068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:03.403299 6068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:03.403359 6068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:04:03.408782 6068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:03.409881 6068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:03.411041 6068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:03.421804 6076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:03.421864 6068 server_base.cc:1034] running on GCE node
W20260810 23:04:03.421842 6077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:03.421804 6079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:03.422587 6068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:03.423324 6068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:03.424521 6068 hybrid_clock.cc:648] HybridClock initialized: now 1786403043424468 us; error 74 us; skew 500 ppm
Aug 10 23:04:03 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:03.427484 6068 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:04:03.428514 6068 webserver.cc:460] Webserver started at http://127.15.56.2:41759/ using document root <none> and password file <none>
I20260810 23:04:03.429136 6068 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:03.429214 6068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:03.429440 6068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:03.431772 6068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "16f5678be0b141698d45e5d6a5c2ae77"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "5c9b625989ff1c1f1101cd821140426c"
server_key_iv: "ec406d663dfb4c9ddfe5c4026d7bf0ea"
server_key_version: "encryptionkey@0"
I20260810 23:04:03.432397 6068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "16f5678be0b141698d45e5d6a5c2ae77"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "5c9b625989ff1c1f1101cd821140426c"
server_key_iv: "ec406d663dfb4c9ddfe5c4026d7bf0ea"
server_key_version: "encryptionkey@0"
I20260810 23:04:03.436493 6068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 23:04:03.439009 6088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:03.440347 6068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:04:03.440474 6068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "16f5678be0b141698d45e5d6a5c2ae77"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "5c9b625989ff1c1f1101cd821140426c"
server_key_iv: "ec406d663dfb4c9ddfe5c4026d7bf0ea"
server_key_version: "encryptionkey@0"
I20260810 23:04:03.440578 6068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:03.473613 6068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:03.477101 6068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:03.477308 6068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:03.477990 6068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:03.478911 6068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:03.478962 6068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:03.479017 6068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:03.479046 6068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:03.488245 6068 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:37171
I20260810 23:04:03.488305 6255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:37171 every 8 connection(s)
I20260810 23:04:03.489305 6068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:04:03.497908 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 6068
I20260810 23:04:03.498059 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:04:03.498311 15584 external_mini_cluster.cc:1428] Setting key 76b14873a3d536353b2be7a83b6a6846
Aug 10 23:04:03 dist-test-slave-6bbx krb5kdc[5717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:03.504984 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.491348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:37951 (local address 127.15.56.62:44547)
0810 23:04:03.491474 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.491477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.492518 (+ 1041us) server_negotiation.cc:407] Connection header received
0810 23:04:03.494049 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.494053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.494113 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.494194 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:03.495497 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.496074 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.497346 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.497568 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.497662 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.500671 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:03.500675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:03.500677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:03.500710 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:03.502490 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:03.503075 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:03.503079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:03.503081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:03.503202 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:03.503634 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:03.503638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:03.503640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:03.503939 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:03.504044 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:03.504642 (+ 598us) server_negotiation.cc:299] Negotiation successful
0810 23:04:03.504803 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:04:03.505935 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.491856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:37951)
0810 23:04:03.492356 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.492405 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:03.493817 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:03.494404 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:03.494412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:03.494823 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:03.495343 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.495358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.496204 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.496207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.497218 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.497229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.497725 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.497730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.497935 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.498561 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:03.498584 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:03.500456 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:03.502642 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:03.502648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:03.502666 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:03.502973 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:03.503340 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:03.503343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:03.503345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:03.503526 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:03.504057 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:03.504064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:03.504376 (+ 312us) client_negotiation.cc:769] Sending connection context
0810 23:04:03.504558 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 23:04:03.504883 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260810 23:04:03.507239 6256 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44547
I20260810 23:04:03.507567 6256 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:03.508376 6256 heartbeater.cc:511] Master 127.15.56.62:44547 requested a full tablet report, sending...
I20260810 23:04:03.509846 5784 ts_manager.cc:194] Registered new tserver with Master: 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171)
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:04:03.510457 5784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:37951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:04:03.531498 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44547
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:03.650478 6266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:03.650820 6266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:03.650887 6266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:03.655282 6266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:03.655372 6266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:03.655458 6266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:04:03.661760 6266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:03.663092 6266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:03.664261 6266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:03.674337 6274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:03.674331 6271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:03.674422 6266 server_base.cc:1034] running on GCE node
W20260810 23:04:03.674333 6272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:03.675072 6266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:03.675814 6266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:03.676996 6266 hybrid_clock.cc:648] HybridClock initialized: now 1786403043676975 us; error 34 us; skew 500 ppm
Aug 10 23:04:03 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:03.679955 6266 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:04:03.680872 6266 webserver.cc:460] Webserver started at http://127.15.56.3:34287/ using document root <none> and password file <none>
I20260810 23:04:03.681440 6266 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:03.681504 6266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:03.681689 6266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:03.683900 6266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "7e8f7162548d4c488ec25834fa03afb0"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "0e57d9edffaeb261477e39f4df86fc78"
server_key_iv: "020d1c27e7b1b6d0148dabf90597c443"
server_key_version: "encryptionkey@0"
I20260810 23:04:03.684448 6266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "7e8f7162548d4c488ec25834fa03afb0"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "0e57d9edffaeb261477e39f4df86fc78"
server_key_iv: "020d1c27e7b1b6d0148dabf90597c443"
server_key_version: "encryptionkey@0"
I20260810 23:04:03.688446 6266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:04:03.690989 6282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:03.692319 6266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260810 23:04:03.692427 6266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "7e8f7162548d4c488ec25834fa03afb0"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "0e57d9edffaeb261477e39f4df86fc78"
server_key_iv: "020d1c27e7b1b6d0148dabf90597c443"
server_key_version: "encryptionkey@0"
I20260810 23:04:03.692533 6266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:03.723865 6266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:03.727815 6266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:03.728011 6266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:03.728668 6266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:03.729588 6266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:03.729637 6266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:03.729681 6266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:03.729710 6266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:03.738660 6266 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:43117
I20260810 23:04:03.738698 6443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:43117 every 8 connection(s)
I20260810 23:04:03.739626 6266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:04:03.749821 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 6266
I20260810 23:04:03.750001 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
Aug 10 23:04:03 dist-test-slave-6bbx krb5kdc[5717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403043, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:03.750312 15584 external_mini_cluster.cc:1428] Setting key 247df3c7d584984b6d5413def5acd652
I20260810 23:04:03.755326 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.741603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:60397 (local address 127.15.56.62:44547)
0810 23:04:03.741746 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.741750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.742668 (+ 918us) server_negotiation.cc:407] Connection header received
0810 23:04:03.744248 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.744260 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.744329 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.744401 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:03.745782 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.746333 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.747521 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.747784 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.747880 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.751071 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:03.751076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:03.751078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:03.751114 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:03.752770 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:03.753422 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:03.753425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:03.753427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:03.753514 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:03.753977 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:03.753980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:03.753982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:03.754252 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:03.754344 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:03.754971 (+ 627us) server_negotiation.cc:299] Negotiation successful
0810 23:04:03.755090 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:04:03.756295 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.741939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:60397)
0810 23:04:03.742473 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.742525 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:03.744006 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:03.744610 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:03.744619 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:03.745039 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:03.745614 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.745630 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.746466 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.746468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.747386 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.747397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.747947 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.747952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.748131 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.748787 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:03.748806 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:03.750785 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:03.752933 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:03.752941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:03.752963 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:03.753321 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:03.753656 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:03.753659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:03.753660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:03.753854 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:03.754382 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:03.754389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:03.754703 (+ 314us) client_negotiation.cc:769] Sending connection context
0810 23:04:03.754887 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 23:04:03.755245 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":131,"threads_started":1}
I20260810 23:04:03.757814 6444 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44547
I20260810 23:04:03.758081 6444 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:03.758668 6444 heartbeater.cc:511] Master 127.15.56.62:44547 requested a full tablet report, sending...
I20260810 23:04:03.759852 5784 ts_manager.cc:194] Registered new tserver with Master: 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117)
I20260810 23:04:03.760468 5784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:60397
I20260810 23:04:03.765260 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:04:03.775390 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.767382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.15.56.62:44547)
0810 23:04:03.767525 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.767528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.767658 (+ 130us) server_negotiation.cc:407] Connection header received
0810 23:04:03.767838 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.767840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.767885 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.767947 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:03.768785 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.769357 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.770411 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.770646 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.770748 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.772288 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:03.772292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:03.772295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:03.772331 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:03.773761 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:03.774176 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:03.774179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:03.774207 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:03.774297 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:03.774577 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:03.774579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:03.774580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:03.774814 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:03.774918 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:03.775130 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 23:04:03.775259 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:04:03.779003 5784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:04:03.782366 5784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:04:03.798765 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.791993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:43117 (local address 127.0.0.1:45966)
0810 23:04:03.793221 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.793284 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:03.793443 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:03.793964 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:03.793970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:03.793992 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:03.794147 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.794155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.795413 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.795416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.797014 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.797025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.798157 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.798161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.798282 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.798332 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 23:04:03.798444 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 23:04:03.798594 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":144,"thread_start_us":90,"threads_started":1}
I20260810 23:04:03.798887 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.792869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.15.56.1:40503)
0810 23:04:03.793270 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.793277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.793585 (+ 308us) server_negotiation.cc:407] Connection header received
0810 23:04:03.793673 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.793678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.793886 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.794028 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:03.794462 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.795655 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.797668 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.798305 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.798419 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.798565 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:03.798654 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 23:04:03.798755 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":113,"threads_started":1}
I20260810 23:04:03.799291 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.792214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.15.56.3:43117)
0810 23:04:03.792849 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.792866 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.793384 (+ 518us) server_negotiation.cc:407] Connection header received
0810 23:04:03.793620 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.793626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.793809 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.793961 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:03.794263 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.795246 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.797136 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.798001 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.798119 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.798304 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:03.799013 (+ 709us) server_negotiation.cc:299] Negotiation successful
0810 23:04:03.799134 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":153,"threads_started":1}
I20260810 23:04:03.798914 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.792052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:40503 (local address 127.0.0.1:45140)
0810 23:04:03.793286 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.793297 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:03.793434 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:03.794201 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:03.794204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:03.794224 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:03.794335 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.794343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.795844 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.795848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.797518 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.797527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.798420 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.798424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.798525 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.798551 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:04:03.798624 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 23:04:03.798699 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":164,"thread_start_us":125,"threads_started":1}
I20260810 23:04:03.799710 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.792869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.15.56.2:37171)
0810 23:04:03.793760 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.793772 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.793793 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:04:03.793861 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.793866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.794026 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.794165 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:03.794528 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.795775 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.797613 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.798434 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.798534 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.798674 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:03.799414 (+ 740us) server_negotiation.cc:299] Negotiation successful
0810 23:04:03.799522 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":115,"threads_started":1}
I20260810 23:04:03.799307 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.792308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:37171 (local address 127.0.0.1:53816)
0810 23:04:03.793401 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.793415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:03.793505 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:03.794280 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:03.794283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:03.794302 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:03.794398 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.794404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.795928 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.795931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.797470 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.797484 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.798735 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.798739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.798840 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.798862 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:04:03.798929 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 23:04:03.798997 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"mutex_wait_us":88,"thread_start_us":86,"threads_started":1}
I20260810 23:04:03.803113 6356 tablet_service.cc:1467] Processing CreateTablet for tablet a52d5fdeb2274f3789417af0bcc3b028 (DEFAULT_TABLE table=test-table [id=3b3595d2cd634700b9892aa4a6857e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:03.803360 5962 tablet_service.cc:1467] Processing CreateTablet for tablet a52d5fdeb2274f3789417af0bcc3b028 (DEFAULT_TABLE table=test-table [id=3b3595d2cd634700b9892aa4a6857e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:03.803607 6161 tablet_service.cc:1467] Processing CreateTablet for tablet a52d5fdeb2274f3789417af0bcc3b028 (DEFAULT_TABLE table=test-table [id=3b3595d2cd634700b9892aa4a6857e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:03.804405 5962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52d5fdeb2274f3789417af0bcc3b028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:03.804417 6356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52d5fdeb2274f3789417af0bcc3b028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:03.804706 6161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52d5fdeb2274f3789417af0bcc3b028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:03.812194 6468 tablet_bootstrap.cc:492] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Bootstrap starting.
I20260810 23:04:03.812816 6469 tablet_bootstrap.cc:492] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0: Bootstrap starting.
I20260810 23:04:03.813874 6476 tablet_bootstrap.cc:492] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77: Bootstrap starting.
I20260810 23:04:03.814926 6468 tablet_bootstrap.cc:654] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:03.815274 6469 tablet_bootstrap.cc:654] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:03.815943 6468 log.cc:824] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:03.816022 6469 log.cc:824] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:03.816679 6476 tablet_bootstrap.cc:654] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:03.817611 6476 log.cc:824] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:03.818136 6469 tablet_bootstrap.cc:492] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0: No bootstrap required, opened a new log
I20260810 23:04:03.818212 6468 tablet_bootstrap.cc:492] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: No bootstrap required, opened a new log
I20260810 23:04:03.818361 6469 ts_tablet_manager.cc:1397] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:04:03.818415 6468 ts_tablet_manager.cc:1397] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:04:03.819705 6476 tablet_bootstrap.cc:492] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77: No bootstrap required, opened a new log
I20260810 23:04:03.819887 6476 ts_tablet_manager.cc:1397] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:04:03.821400 6469 raft_consensus.cc:348] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } }
I20260810 23:04:03.821624 6469 raft_consensus.cc:374] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:03.821655 6469 raft_consensus.cc:729] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8f7162548d4c488ec25834fa03afb0, State: Initialized, Role: FOLLOWER
I20260810 23:04:03.822093 6469 consensus_queue.cc:260] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } }
I20260810 23:04:03.823403 6444 heartbeater.cc:503] Master 127.15.56.62:44547 was elected leader, sending a full tablet report...
I20260810 23:04:03.823730 6476 raft_consensus.cc:348] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } }
I20260810 23:04:03.824008 6476 raft_consensus.cc:374] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:03.824115 6476 raft_consensus.cc:729] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5678be0b141698d45e5d6a5c2ae77, State: Initialized, Role: FOLLOWER
I20260810 23:04:03.822992 6468 raft_consensus.cc:348] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } }
I20260810 23:04:03.824536 6469 ts_tablet_manager.cc:1428] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:04:03.824618 6476 consensus_queue.cc:260] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } }
I20260810 23:04:03.824895 6468 raft_consensus.cc:374] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:03.825019 6468 raft_consensus.cc:729] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71725911b2a44f989f68a583ba8a0004, State: Initialized, Role: FOLLOWER
I20260810 23:04:03.825364 6256 heartbeater.cc:503] Master 127.15.56.62:44547 was elected leader, sending a full tablet report...
I20260810 23:04:03.825551 6468 consensus_queue.cc:260] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } }
I20260810 23:04:03.825766 6476 ts_tablet_manager.cc:1428] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:04:03.826342 6061 heartbeater.cc:503] Master 127.15.56.62:44547 was elected leader, sending a full tablet report...
I20260810 23:04:03.826573 6468 ts_tablet_manager.cc:1428] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 23:04:03.845860 6482 raft_consensus.cc:482] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:04:03.846097 6482 raft_consensus.cc:504] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } }
I20260810 23:04:03.847306 6482 leader_election.cc:288] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117), 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171)
I20260810 23:04:03.851428 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.847755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:59535 (local address 127.15.56.2:37171)
0810 23:04:03.847888 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.847892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.848036 (+ 144us) server_negotiation.cc:407] Connection header received
0810 23:04:03.848182 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.848185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.848231 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.848300 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:03.848610 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.849284 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.850454 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.850872 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.850972 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.851044 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:03.851221 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 23:04:03.851309 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:04:03.851452 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.847583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:37171 (local address 127.15.56.1:59535)
0810 23:04:03.847939 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.847952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:03.848074 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:03.848327 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:03.848330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:03.848346 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:03.848480 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.848486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.849370 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.849373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.850316 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.850327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.850996 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.850999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.851098 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.851120 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:04:03.851188 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:04:03.851271 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":22,"thread_start_us":172,"threads_started":1}
I20260810 23:04:03.851485 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.847643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:41945 (local address 127.15.56.3:43117)
0810 23:04:03.847835 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.847840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.848271 (+ 431us) server_negotiation.cc:407] Connection header received
0810 23:04:03.848412 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.848415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.848465 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.848537 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:03.848846 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.849416 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.850486 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.850908 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.851009 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.851077 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:03.851261 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 23:04:03.851348 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 23:04:03.851501 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.847584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:43117 (local address 127.15.56.1:41945)
0810 23:04:03.848166 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.848177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:03.848302 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:03.848564 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:03.848567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:03.848584 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:03.848687 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.848693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.849523 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.849527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.850355 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.850365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.851026 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.851029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.851138 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.851158 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 23:04:03.851226 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:04:03.851354 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":27,"thread_start_us":315,"threads_started":1}
I20260810 23:04:03.852247 6382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52d5fdeb2274f3789417af0bcc3b028" candidate_uuid: "71725911b2a44f989f68a583ba8a0004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8f7162548d4c488ec25834fa03afb0" is_pre_election: true
I20260810 23:04:03.852247 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52d5fdeb2274f3789417af0bcc3b028" candidate_uuid: "71725911b2a44f989f68a583ba8a0004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5678be0b141698d45e5d6a5c2ae77" is_pre_election: true
I20260810 23:04:03.852658 6187 raft_consensus.cc:2454] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71725911b2a44f989f68a583ba8a0004 in term 0.
I20260810 23:04:03.852658 6382 raft_consensus.cc:2454] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71725911b2a44f989f68a583ba8a0004 in term 0.
I20260810 23:04:03.853195 5892 leader_election.cc:302] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71725911b2a44f989f68a583ba8a0004, 7e8f7162548d4c488ec25834fa03afb0; no voters:
I20260810 23:04:03.853502 6482 raft_consensus.cc:2781] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:04:03.853601 6482 raft_consensus.cc:482] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:04:03.853652 6482 raft_consensus.cc:3037] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:03.854786 6482 raft_consensus.cc:504] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } }
I20260810 23:04:03.855190 6482 leader_election.cc:288] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 1 election: Requested vote from peers 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117), 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171)
I20260810 23:04:03.855607 6382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52d5fdeb2274f3789417af0bcc3b028" candidate_uuid: "71725911b2a44f989f68a583ba8a0004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8f7162548d4c488ec25834fa03afb0"
I20260810 23:04:03.855732 6187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52d5fdeb2274f3789417af0bcc3b028" candidate_uuid: "71725911b2a44f989f68a583ba8a0004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5678be0b141698d45e5d6a5c2ae77"
I20260810 23:04:03.855803 6382 raft_consensus.cc:3037] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:03.855878 6187 raft_consensus.cc:3037] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:03.857103 6187 raft_consensus.cc:2454] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71725911b2a44f989f68a583ba8a0004 in term 1.
I20260810 23:04:03.857160 6382 raft_consensus.cc:2454] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71725911b2a44f989f68a583ba8a0004 in term 1.
I20260810 23:04:03.857420 5894 leader_election.cc:302] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5678be0b141698d45e5d6a5c2ae77, 71725911b2a44f989f68a583ba8a0004; no voters:
I20260810 23:04:03.857616 6482 raft_consensus.cc:2781] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:03.857839 6482 raft_consensus.cc:686] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 1 LEADER]: Becoming Leader. State: Replica: 71725911b2a44f989f68a583ba8a0004, State: Running, Role: LEADER
I20260810 23:04:03.858210 6482 consensus_queue.cc:237] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } }
I20260810 23:04:03.862299 5784 catalog_manager.cc:5495] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71725911b2a44f989f68a583ba8a0004 (127.15.56.1). New cstate: current_term: 1 leader_uuid: "71725911b2a44f989f68a583ba8a0004" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } health_report { overall_health: UNKNOWN } } }
I20260810 23:04:03.887933 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.883940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.15.56.1:40503)
0810 23:04:03.884099 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.884104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.884121 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:04:03.884316 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.884319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.884384 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.884486 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:04:03.884746 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.885259 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.886143 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.886368 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.886465 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.886817 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:04:03.887306 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:04:03.887436 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:03.887511 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 23:04:03.887587 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:04:03.895223 6187 raft_consensus.cc:1261] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77 [term 1 FOLLOWER]: Refusing update from remote peer 71725911b2a44f989f68a583ba8a0004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:03.895223 6382 raft_consensus.cc:1261] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0 [term 1 FOLLOWER]: Refusing update from remote peer 71725911b2a44f989f68a583ba8a0004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:03.896070 6485 consensus_queue.cc:1035] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:03.896353 6482 consensus_queue.cc:1035] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:03.906069 6491 mvcc.cc:205] Tried to move back new op lower bound from 7317106867785904128 to 7317106867646271488. Current Snapshot: MvccSnapshot[applied={T|T < 7317106867785904128}]
I20260810 23:04:03.915935 5783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:03.916139 5783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:03.920287 5783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Sending DeleteTablet for 3 replicas of tablet a52d5fdeb2274f3789417af0bcc3b028
I20260810 23:04:03.921304 5962 tablet_service.cc:1514] Processing DeleteTablet for tablet a52d5fdeb2274f3789417af0bcc3b028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:03 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45140
I20260810 23:04:03.921306 6356 tablet_service.cc:1514] Processing DeleteTablet for tablet a52d5fdeb2274f3789417af0bcc3b028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:03 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45966
I20260810 23:04:03.921464 6161 tablet_service.cc:1514] Processing DeleteTablet for tablet a52d5fdeb2274f3789417af0bcc3b028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:03 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:53816
I20260810 23:04:03.922186 6502 tablet_replica.cc:330] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: stopping tablet replica
I20260810 23:04:03.922446 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 5874
I20260810 23:04:03.938124 6505 tablet_replica.cc:330] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77: stopping tablet replica
I20260810 23:04:03.938572 6505 raft_consensus.cc:2229] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:03.938997 6505 raft_consensus.cc:2258] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:03.941581 6503 tablet_replica.cc:330] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0: stopping tablet replica
I20260810 23:04:03.941987 6503 raft_consensus.cc:2229] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:03.942392 6503 raft_consensus.cc:2258] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:03.942934 6505 ts_tablet_manager.cc:1905] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:03.944561 6503 ts_tablet_manager.cc:1905] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:03.946208 6505 ts_tablet_manager.cc:1918] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:03.946429 6505 log.cc:1196] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/a52d5fdeb2274f3789417af0bcc3b028
I20260810 23:04:03.947460 6503 ts_tablet_manager.cc:1918] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:03.947567 6503 log.cc:1196] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/a52d5fdeb2274f3789417af0bcc3b028
I20260810 23:04:03.947938 6503 ts_tablet_manager.cc:1939] T a52d5fdeb2274f3789417af0bcc3b028 P 7e8f7162548d4c488ec25834fa03afb0: Deleting consensus metadata
I20260810 23:04:03.949419 6505 ts_tablet_manager.cc:1939] T a52d5fdeb2274f3789417af0bcc3b028 P 16f5678be0b141698d45e5d6a5c2ae77: Deleting consensus metadata
I20260810 23:04:03.949476 5765 catalog_manager.cc:4841] TS 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117): tablet a52d5fdeb2274f3789417af0bcc3b028 (table test-table [id=3b3595d2cd634700b9892aa4a6857e74]) successfully deleted
W20260810 23:04:03.950620 5770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:40503 (error 108)
I20260810 23:04:03.950876 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 6068
I20260810 23:04:03.974860 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.974500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:40503 (local address 127.0.0.1:45158)
0810 23:04:03.974653 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.974751 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 23:04:03.975181 5770 catalog_manager.cc:4584] TS 71725911b2a44f989f68a583ba8a0004 (127.15.56.1:40503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a52d5fdeb2274f3789417af0bcc3b028: Network error: Client connection negotiation failed: client connection to 127.15.56.1:40503: connect: Connection refused (error 111)
I20260810 23:04:03.979776 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 6266
I20260810 23:04:03.980257 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.979897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:37171 (local address 127.0.0.1:53830)
0810 23:04:03.980052 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.980153 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.2:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:04:04.002516 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 5738
I20260810 23:04:04.019989 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44547
--webserver_interface=127.15.56.62
--webserver_port=32949
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:44547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:04.140863 6507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:04.141187 6507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:04.141229 6507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:04.145354 6507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:04:04.145428 6507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:04.145447 6507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:04.145463 6507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:04:04.145478 6507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:04:04.151094 6507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:44547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:04.152299 6507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:04.153366 6507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:04.168229 6514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:04.168229 6516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:04.168231 6513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:04.168671 6507 server_base.cc:1034] running on GCE node
I20260810 23:04:04.169376 6507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:04.170692 6507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:04.171875 6507 hybrid_clock.cc:648] HybridClock initialized: now 1786403044171846 us; error 39 us; skew 500 ppm
Aug 10 23:04:04 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403044, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:04.175331 6507 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:04:04.176461 6507 webserver.cc:460] Webserver started at http://127.15.56.62:32949/ using document root <none> and password file <none>
I20260810 23:04:04.177120 6507 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:04.177222 6507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:04.182159 6507 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:04:04.184927 6526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:04.186451 6507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:04:04.186618 6507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "c8ffc07682a24ae2b5da1d1b1db3003e"
format_stamp: "Formatted at 2026-08-10 23:04:02 on dist-test-slave-6bbx"
server_key: "7124602168ea99fce4d2af84347c31a4"
server_key_iv: "3bbfc056c10a7a80d7f2467920425f26"
server_key_version: "encryptionkey@0"
I20260810 23:04:04.187305 6507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:04.206064 6507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:04.209829 6507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:04.210055 6507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:04.218511 6507 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:44547
I20260810 23:04:04.218792 6600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:44547 every 8 connection(s)
I20260810 23:04:04.219921 6507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:04:04.225138 6601 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:04:04.227175 6601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Bootstrap starting.
I20260810 23:04:04.228148 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 6507
I20260810 23:04:04.228641 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:40503
--local_ip_for_outbound_sockets=127.15.56.1
--tserver_master_addrs=127.15.56.62:44547
--webserver_port=41723
--webserver_interface=127.15.56.1
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:04:04.246850 6601 log.cc:824] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:04.257833 6601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:04:04.258311 6601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Bootstrap complete.
I20260810 23:04:04.263216 6601 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:04.263774 6601 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ffc07682a24ae2b5da1d1b1db3003e, State: Initialized, Role: FOLLOWER
I20260810 23:04:04.264390 6601 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:04.264535 6601 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:04:04.264613 6601 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:04:04.264719 6601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:04:04.267047 6601 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:04.267409 6601 leader_election.cc:302] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8ffc07682a24ae2b5da1d1b1db3003e; no voters:
I20260810 23:04:04.267793 6601 leader_election.cc:288] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:04:04.267969 6610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:04:04.268220 6610 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 2 LEADER]: Becoming Leader. State: Replica: c8ffc07682a24ae2b5da1d1b1db3003e, State: Running, Role: LEADER
I20260810 23:04:04.268537 6610 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:04.269228 6601 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:04:04.270190 6615 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8ffc07682a24ae2b5da1d1b1db3003e. Latest consensus state: current_term: 2 leader_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } } }
I20260810 23:04:04.270197 6613 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } } }
I20260810 23:04:04.270344 6615 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:04.270351 6613 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:04.274293 6632 catalog_manager.cc:1259] Loaded cluster ID: ac64f1c9168e437c9fe7a20143006f40
I20260810 23:04:04.274371 6632 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: loading cluster ID for follower catalog manager: success
W20260810 23:04:04.275213 6632 catalog_manager.cc:1576] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260810 23:04:04.275276 6632 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260810 23:04:04.275341 6626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:04:04.277966 6626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:04:04.278509 6626 catalog_manager.cc:1259] Loaded cluster ID: ac64f1c9168e437c9fe7a20143006f40
I20260810 23:04:04.278580 6626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:04:04.279631 6626 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 23:04:04.284060 6626 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 23:04:04.289836 6626 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:04:04.290693 6626 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Loaded TSK: 0
I20260810 23:04:04.292054 6626 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:04:04.355203 6607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:04.355500 6607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:04.355544 6607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:04.359768 6607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:04.359855 6607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:04.359918 6607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:04:04.365720 6607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:40503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=41723
--tserver_master_addrs=127.15.56.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:04.366959 6607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:04.368090 6607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:04.378727 6645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:04.378727 6644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:04.379001 6648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:04.379168 6607 server_base.cc:1034] running on GCE node
I20260810 23:04:04.379786 6607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:04.380674 6607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:04.381907 6607 hybrid_clock.cc:648] HybridClock initialized: now 1786403044381870 us; error 47 us; skew 500 ppm
Aug 10 23:04:04 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403044, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:04.385814 6607 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:04:04.387221 6607 webserver.cc:460] Webserver started at http://127.15.56.1:41723/ using document root <none> and password file <none>
I20260810 23:04:04.388062 6607 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:04.388185 6607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:04.393991 6607 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:04:04.397185 6656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:04.399029 6607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:04:04.399204 6607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "71725911b2a44f989f68a583ba8a0004"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "06bdc983af5ce1a6ff10ab211bdfeccd"
server_key_iv: "27b6798194683e42142149919290c741"
server_key_version: "encryptionkey@0"
I20260810 23:04:04.400058 6607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:04.423750 6607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:04.427637 6607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:04.427902 6607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:04.428685 6607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:04.430024 6666 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:04:04.434435 6607 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:04:04.434506 6607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 23:04:04.434559 6607 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:04:04.436407 6607 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:04:04.436455 6607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260810 23:04:04.436765 6666 tablet_bootstrap.cc:492] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Bootstrap starting.
I20260810 23:04:04.446038 6607 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:40503
I20260810 23:04:04.446103 6819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:40503 every 8 connection(s)
I20260810 23:04:04.447113 6607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:04:04.447476 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 6607
I20260810 23:04:04.448004 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:37171
--local_ip_for_outbound_sockets=127.15.56.2
--tserver_master_addrs=127.15.56.62:44547
--webserver_port=41759
--webserver_interface=127.15.56.2
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:04:04.457110 6666 log.cc:824] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:04:04 dist-test-slave-6bbx krb5kdc[5717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403044, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:04.464938 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.449631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:60979)
0810 23:04:04.450517 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:04.450579 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:04.451423 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:04.452783 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:04.452792 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:04.453237 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:04.453718 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.453734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.455094 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.455097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.456080 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.456092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.457051 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.457055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.457269 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.457931 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:04.457952 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:04.459423 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:04.461642 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:04.461648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:04.461663 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:04.462005 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:04.462447 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:04.462449 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:04.462451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:04.462578 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:04.463174 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:04.463181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:04.463431 (+ 250us) client_negotiation.cc:769] Sending connection context
0810 23:04:04.463597 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 23:04:04.463986 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":145,"threads_started":1}
I20260810 23:04:04.464926 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.449610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:60979 (local address 127.15.56.62:44547)
0810 23:04:04.450184 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.450205 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.451728 (+ 1523us) server_negotiation.cc:407] Connection header received
0810 23:04:04.452016 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.452027 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.452361 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.452681 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:04.453911 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.454921 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.456209 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.456889 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.457063 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.459751 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:04.459758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:04.459773 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:04.459805 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:04.461493 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:04.462160 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:04.462167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:04.462173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:04.462343 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:04.462671 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:04.462674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:04.462676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:04.463057 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:04.463184 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:04.463791 (+ 607us) server_negotiation.cc:299] Negotiation successful
0810 23:04:04.464068 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":131,"threads_started":1}
I20260810 23:04:04.467180 6666 tablet_bootstrap.cc:492] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:04:04.467628 6666 tablet_bootstrap.cc:492] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Bootstrap complete.
I20260810 23:04:04.467851 6820 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44547
I20260810 23:04:04.468086 6820 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:04.468618 6820 heartbeater.cc:511] Master 127.15.56.62:44547 requested a full tablet report, sending...
I20260810 23:04:04.469153 6666 ts_tablet_manager.cc:1397] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.004s
I20260810 23:04:04.471138 6547 ts_manager.cc:194] Registered new tserver with Master: 71725911b2a44f989f68a583ba8a0004 (127.15.56.1:40503)
I20260810 23:04:04.472148 6666 raft_consensus.cc:348] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } }
I20260810 23:04:04.472683 6666 raft_consensus.cc:729] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71725911b2a44f989f68a583ba8a0004, State: Initialized, Role: FOLLOWER
I20260810 23:04:04.473187 6666 consensus_queue.cc:260] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } } peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } }
I20260810 23:04:04.474323 6666 ts_tablet_manager.cc:1428] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:04:04.474895 6547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:60979
I20260810 23:04:04.477624 6820 heartbeater.cc:503] Master 127.15.56.62:44547 was elected leader, sending a full tablet report...
Aug 10 23:04:04 dist-test-slave-6bbx krb5kdc[5717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403044, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:04:04.486068 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.474234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:40503 (local address 127.0.0.1:45164)
0810 23:04:04.475406 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:04.475464 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:04.475618 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:04.476233 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:04.476240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:04.476709 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:04.476851 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.476860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.478276 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.478279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.479217 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.479228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.480058 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.480061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.480154 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.480720 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:04.480731 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:04.482319 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:04.484382 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:04.484401 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:04.484405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:04.484654 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:04.484950 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:04.484952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:04.484954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:04.485032 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:04.485518 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:04.485523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:04.485634 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 23:04:04.485749 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 23:04:04.485923 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":102,"threads_started":1}
I20260810 23:04:04.486193 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.474817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.15.56.1:40503)
0810 23:04:04.475135 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.475146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.475574 (+ 428us) server_negotiation.cc:407] Connection header received
0810 23:04:04.475845 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.475851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.476062 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.476195 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:04.476990 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.478180 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.479330 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.479916 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.480026 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.482461 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:04.482471 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:04.482485 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:04.482513 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:04.484244 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:04.484763 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:04.484770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:04.484772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:04.484868 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:04.485101 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:04.485104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:04.485105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:04.485411 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:04.485534 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:04.485848 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 23:04:04.486026 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
I20260810 23:04:04.487239 6724 tablet_service.cc:1514] Processing DeleteTablet for tablet a52d5fdeb2274f3789417af0bcc3b028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:03 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45164
I20260810 23:04:04.488125 6830 tablet_replica.cc:330] stopping tablet replica
I20260810 23:04:04.488430 6830 raft_consensus.cc:2229] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:04.488677 6830 raft_consensus.cc:2258] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:04.489704 6830 ts_tablet_manager.cc:1905] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:04.493430 6830 ts_tablet_manager.cc:1918] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:04.493520 6830 log.cc:1196] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal/wals/a52d5fdeb2274f3789417af0bcc3b028
I20260810 23:04:04.493769 6830 ts_tablet_manager.cc:1939] T a52d5fdeb2274f3789417af0bcc3b028 P 71725911b2a44f989f68a583ba8a0004: Deleting consensus metadata
I20260810 23:04:04.495462 6532 catalog_manager.cc:4841] TS 71725911b2a44f989f68a583ba8a0004 (127.15.56.1:40503): tablet a52d5fdeb2274f3789417af0bcc3b028 (table test-table [id=3b3595d2cd634700b9892aa4a6857e74]) successfully deleted
W20260810 23:04:04.580472 6822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:04.580786 6822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:04.580853 6822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:04.584921 6822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:04.584998 6822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:04.585083 6822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:04:04.590556 6822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:37171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=41759
--tserver_master_addrs=127.15.56.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:04.591842 6822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:04.592976 6822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:04.602890 6839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:04.602895 6837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:04.602895 6843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:04.603188 6822 server_base.cc:1034] running on GCE node
I20260810 23:04:04.603647 6822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:04.604353 6822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:04.605576 6822 hybrid_clock.cc:648] HybridClock initialized: now 1786403044605555 us; error 35 us; skew 500 ppm
Aug 10 23:04:04 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403044, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:04.611819 6822 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:04:04.612807 6822 webserver.cc:460] Webserver started at http://127.15.56.2:41759/ using document root <none> and password file <none>
I20260810 23:04:04.613379 6822 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:04.613457 6822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:04.617522 6822 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 23:04:04.619774 6855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:04.621012 6822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:04:04.621120 6822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "16f5678be0b141698d45e5d6a5c2ae77"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "5c9b625989ff1c1f1101cd821140426c"
server_key_iv: "ec406d663dfb4c9ddfe5c4026d7bf0ea"
server_key_version: "encryptionkey@0"
I20260810 23:04:04.621714 6822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:04.638741 6822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:04.641938 6822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:04.642131 6822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:04.642728 6822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:04.643682 6822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:04.643747 6822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:04.643806 6822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:04.643841 6822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:04.652873 6822 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:37171
I20260810 23:04:04.652906 7021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:37171 every 8 connection(s)
I20260810 23:04:04.653951 6822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:04:04.661996 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 6822
I20260810 23:04:04.662456 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:43117
--local_ip_for_outbound_sockets=127.15.56.3
--tserver_master_addrs=127.15.56.62:44547
--webserver_port=34287
--webserver_interface=127.15.56.3
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:04:04 dist-test-slave-6bbx krb5kdc[5717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403044, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:04.668498 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.655886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:47889 (local address 127.15.56.62:44547)
0810 23:04:04.656048 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.656052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.656951 (+ 899us) server_negotiation.cc:407] Connection header received
0810 23:04:04.658367 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.658371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.658426 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.658509 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:04.659953 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.660492 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.661711 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.661960 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.662072 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.664738 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:04.664743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:04.664745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:04.664780 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:04.666362 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:04.666888 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:04.666891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:04.666893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:04.666995 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:04.667367 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:04.667370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:04.667372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:04.667557 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:04.667642 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:04.668185 (+ 543us) server_negotiation.cc:299] Negotiation successful
0810 23:04:04.668329 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:04:04.669376 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.656314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:47889)
0810 23:04:04.656791 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:04.656838 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:04.658131 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:04.658673 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:04.658681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:04.659092 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:04.659609 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.659623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.660625 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.660627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.661564 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.661574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.662193 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.662199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.662374 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.663056 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:04.663075 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:04.664544 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:04.666468 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:04.666475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:04.666490 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:04.666777 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:04.667073 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:04.667076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:04.667078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:04.667266 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:04.667656 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:04.667680 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:04.667941 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 23:04:04.668108 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 23:04:04.668462 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260810 23:04:04.670624 7023 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44547
I20260810 23:04:04.670907 7023 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:04.671414 7023 heartbeater.cc:511] Master 127.15.56.62:44547 requested a full tablet report, sending...
I20260810 23:04:04.672556 6547 ts_manager.cc:194] Registered new tserver with Master: 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171)
I20260810 23:04:04.673344 6547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:47889
W20260810 23:04:04.781478 7026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:04.781771 7026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:04.781816 7026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:04.785877 7026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:04.785948 7026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:04.786008 7026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:04:04.791700 7026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:43117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=34287
--tserver_master_addrs=127.15.56.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:04.792855 7026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:04.793880 7026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:04:04.803988 7026 server_base.cc:1034] running on GCE node
W20260810 23:04:04.804030 7037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:04.804157 7035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:04.804030 7040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:04.804613 7026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:04.805274 7026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:04.806442 7026 hybrid_clock.cc:648] HybridClock initialized: now 1786403044806422 us; error 32 us; skew 500 ppm
Aug 10 23:04:04 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403044, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:04.809369 7026 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:04:04.810427 7026 webserver.cc:460] Webserver started at http://127.15.56.3:34287/ using document root <none> and password file <none>
I20260810 23:04:04.811164 7026 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:04.811256 7026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:04.815981 7026 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:04:04.818332 7052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:04.819793 7026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:04:04.819926 7026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "7e8f7162548d4c488ec25834fa03afb0"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "0e57d9edffaeb261477e39f4df86fc78"
server_key_iv: "020d1c27e7b1b6d0148dabf90597c443"
server_key_version: "encryptionkey@0"
I20260810 23:04:04.820520 7026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:04.847609 7026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:04.851116 7026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:04.851336 7026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:04.852074 7026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:04.853027 7026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:04.853080 7026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:04.853137 7026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:04.853166 7026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:04.862643 7026 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:43117
I20260810 23:04:04.862730 7231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:43117 every 8 connection(s)
I20260810 23:04:04.863838 7026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:04:04.870970 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 7026
Aug 10 23:04:04 dist-test-slave-6bbx krb5kdc[5717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403044, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:04.879166 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.871731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.15.56.62:44547)
0810 23:04:04.871998 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.872002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.872108 (+ 106us) server_negotiation.cc:407] Connection header received
0810 23:04:04.872239 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.872242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.872308 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.872383 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:04.873102 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.873616 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.874707 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.874947 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.875030 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.876317 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:04.876321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:04.876323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:04.876366 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:04.877635 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:04.878053 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:04.878056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:04.878057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:04.878122 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:04.878372 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:04.878375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:04.878376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:04.878547 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:04.878691 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:04.878880 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 23:04:04.878986 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260810 23:04:04.879415 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.865954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:36345 (local address 127.15.56.62:44547)
0810 23:04:04.866096 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.866099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.867102 (+ 1003us) server_negotiation.cc:407] Connection header received
0810 23:04:04.868628 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.868631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.868694 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.868770 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:04.870149 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.870714 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.872273 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.872493 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.872583 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.875707 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:04.875710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:04.875712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:04.875735 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:04.877238 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:04.877752 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:04.877755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:04.877757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:04.877831 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:04.878180 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:04.878183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:04.878185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:04.878367 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:04.878460 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:04.879050 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 23:04:04.879143 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:04:04.880478 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.866348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:36345)
0810 23:04:04.866932 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:04.866983 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:04.868394 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:04.868983 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:04.868991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:04.869402 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:04.869988 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.870003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.870847 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.870851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.872138 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.872149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.872641 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.872645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.872835 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.873405 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:04.873423 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:04.875518 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:04.877362 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:04.877367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:04.877381 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:04.877649 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:04.877912 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:04.877915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:04.877918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:04.878098 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:04.878465 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:04.878470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:04.878763 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 23:04:04.879006 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 23:04:04.879340 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":147,"threads_started":1}
I20260810 23:04:04.881682 7232 heartbeater.cc:347] Connected to a master server at 127.15.56.62:44547
I20260810 23:04:04.881922 7232 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:04.882450 7232 heartbeater.cc:511] Master 127.15.56.62:44547 requested a full tablet report, sending...
I20260810 23:04:04.883412 6547 ts_manager.cc:194] Registered new tserver with Master: 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117)
I20260810 23:04:04.884006 6547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:36345
I20260810 23:04:04.892818 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:04:04.896917 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.893409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.15.56.62:44547)
0810 23:04:04.893522 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.893525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.893692 (+ 167us) server_negotiation.cc:407] Connection header received
0810 23:04:04.893847 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.893849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.893953 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.894034 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:04:04.894218 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.894730 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.895583 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.895847 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.895924 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.896200 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:04:04.896611 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:04:04.896694 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:04.896767 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 23:04:04.896816 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 23:04:04.897521 6547 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:04:04.900885 6547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:04:04.908980 6724 tablet_service.cc:1467] Processing CreateTablet for tablet c6ae0343e6054eb99538e62f228b781b (DEFAULT_TABLE table=test-table [id=8cc4ab9503134f19938713b46476f43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:04.909723 6724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ae0343e6054eb99538e62f228b781b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:04.912664 6666 tablet_bootstrap.cc:492] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: Bootstrap starting.
I20260810 23:04:04.913496 6666 tablet_bootstrap.cc:654] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:04.913944 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.908166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:43117 (local address 127.0.0.1:45982)
0810 23:04:04.908446 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:04.908458 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:04.908541 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:04.909496 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:04.909499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:04.909517 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:04.909614 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.909620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.911087 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.911090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.912281 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.912289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.913437 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.913441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.913543 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.913598 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 23:04:04.913689 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:04:04.913783 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":2,"thread_start_us":126,"threads_started":1}
I20260810 23:04:04.914014 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.908618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.15.56.3:43117)
0810 23:04:04.908992 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.909004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.909040 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:04:04.909116 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.909121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.909332 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.909764 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:04.909924 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.910941 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.912569 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.913310 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.913399 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.913520 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:04.913745 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 23:04:04.913854 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":158,"threads_started":1}
I20260810 23:04:04.914780 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.907879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:37171 (local address 127.0.0.1:53836)
0810 23:04:04.908052 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:04.908068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:04.908167 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:04.909172 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:04.909175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:04.909194 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:04.909356 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.909365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.910787 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.910790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.912096 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.912108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.914363 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.914367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.914478 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.914503 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:04:04.914562 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 23:04:04.914671 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":44}
I20260810 23:04:04.915387 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.908137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.15.56.2:37171)
0810 23:04:04.908639 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.908652 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.908677 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:04:04.908790 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.908797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.909025 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.909184 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:04.909486 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.910648 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.912241 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.914135 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.914970 (+ 835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.915108 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:04.915167 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 23:04:04.915247 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":144,"threads_started":1}
I20260810 23:04:04.915953 6666 tablet_bootstrap.cc:492] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: No bootstrap required, opened a new log
I20260810 23:04:04.916049 6666 ts_tablet_manager.cc:1397] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:04:04.916499 6666 raft_consensus.cc:348] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:04.916641 6666 raft_consensus.cc:374] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:04.916683 6666 raft_consensus.cc:729] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71725911b2a44f989f68a583ba8a0004, State: Initialized, Role: FOLLOWER
I20260810 23:04:04.916860 6666 consensus_queue.cc:260] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:04.917219 6666 ts_tablet_manager.cc:1428] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:04:04.918020 7130 tablet_service.cc:1467] Processing CreateTablet for tablet c6ae0343e6054eb99538e62f228b781b (DEFAULT_TABLE table=test-table [id=8cc4ab9503134f19938713b46476f43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:04.919178 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ae0343e6054eb99538e62f228b781b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:04.918725 6925 tablet_service.cc:1467] Processing CreateTablet for tablet c6ae0343e6054eb99538e62f228b781b (DEFAULT_TABLE table=test-table [id=8cc4ab9503134f19938713b46476f43a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:04.919884 6925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ae0343e6054eb99538e62f228b781b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:04.927016 7254 tablet_bootstrap.cc:492] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0: Bootstrap starting.
I20260810 23:04:04.927016 7252 tablet_bootstrap.cc:492] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77: Bootstrap starting.
I20260810 23:04:04.929617 7254 tablet_bootstrap.cc:654] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:04.929672 7252 tablet_bootstrap.cc:654] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:04.930560 7252 log.cc:824] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:04.930831 7254 log.cc:824] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:04.932778 7254 tablet_bootstrap.cc:492] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0: No bootstrap required, opened a new log
I20260810 23:04:04.932778 7252 tablet_bootstrap.cc:492] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77: No bootstrap required, opened a new log
I20260810 23:04:04.932971 7254 ts_tablet_manager.cc:1397] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:04:04.932972 7252 ts_tablet_manager.cc:1397] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:04:04.936074 7252 raft_consensus.cc:348] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:04.936058 7254 raft_consensus.cc:348] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:04.936352 7252 raft_consensus.cc:374] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:04.936354 7254 raft_consensus.cc:374] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:04.936410 7252 raft_consensus.cc:729] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5678be0b141698d45e5d6a5c2ae77, State: Initialized, Role: FOLLOWER
I20260810 23:04:04.936421 7254 raft_consensus.cc:729] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8f7162548d4c488ec25834fa03afb0, State: Initialized, Role: FOLLOWER
I20260810 23:04:04.936861 7252 consensus_queue.cc:260] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:04.936859 7254 consensus_queue.cc:260] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:04.937537 7023 heartbeater.cc:503] Master 127.15.56.62:44547 was elected leader, sending a full tablet report...
I20260810 23:04:04.937537 7232 heartbeater.cc:503] Master 127.15.56.62:44547 was elected leader, sending a full tablet report...
I20260810 23:04:04.937934 7252 ts_tablet_manager.cc:1428] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:04:04.937934 7254 ts_tablet_manager.cc:1428] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260810 23:04:04.948623 6821 tablet.cc:2369] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:04:04.996294 7261 raft_consensus.cc:482] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:04:04.996562 7261 raft_consensus.cc:504] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:04.997817 7261 leader_election.cc:288] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117), 71725911b2a44f989f68a583ba8a0004 (127.15.56.1:40503)
I20260810 23:04:05.002195 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.998298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:60811 (local address 127.15.56.3:43117)
0810 23:04:04.998451 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.998455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.998770 (+ 315us) server_negotiation.cc:407] Connection header received
0810 23:04:04.998878 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.998881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.998931 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.998990 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:04.999239 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.000008 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.001200 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.001620 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.001717 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:05.001785 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:05.001970 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 23:04:05.002045 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 23:04:05.002228 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.998156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:43117 (local address 127.15.56.2:60811)
0810 23:04:04.998658 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:04.998670 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:04.998769 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:04.999028 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:04.999031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:04.999048 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:04.999139 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.999144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.000140 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.000143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:05.001063 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:05.001074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.001746 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.001750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:05.001851 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:05.001871 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 23:04:05.001938 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 23:04:05.002021 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":198,"thread_start_us":118,"threads_started":1}
I20260810 23:04:05.002252 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.998156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:40503 (local address 127.15.56.2:59513)
0810 23:04:04.998316 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:04.998334 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:04.998448 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:04.998835 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:04.998837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:04.998854 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:04.998964 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.998969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.000095 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.000099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:05.001067 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:05.001078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.001802 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.001805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:05.001921 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:05.001943 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:04:05.002007 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 23:04:05.002084 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":72}
I20260810 23:04:05.002254 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.998272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:59513 (local address 127.15.56.1:40503)
0810 23:04:04.998570 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.998574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.998586 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:04:04.998637 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.998639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.998693 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.998807 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:04.999082 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.999968 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.001213 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.001677 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.001797 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:05.001986 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:05.002026 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 23:04:05.002118 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":107,"threads_started":1}
I20260810 23:04:05.002991 7159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ae0343e6054eb99538e62f228b781b" candidate_uuid: "16f5678be0b141698d45e5d6a5c2ae77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8f7162548d4c488ec25834fa03afb0" is_pre_election: true
I20260810 23:04:05.002979 6759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ae0343e6054eb99538e62f228b781b" candidate_uuid: "16f5678be0b141698d45e5d6a5c2ae77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71725911b2a44f989f68a583ba8a0004" is_pre_election: true
I20260810 23:04:05.003340 7159 raft_consensus.cc:2454] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16f5678be0b141698d45e5d6a5c2ae77 in term 0.
I20260810 23:04:05.003340 6759 raft_consensus.cc:2454] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16f5678be0b141698d45e5d6a5c2ae77 in term 0.
I20260810 23:04:05.003931 6856 leader_election.cc:302] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5678be0b141698d45e5d6a5c2ae77, 7e8f7162548d4c488ec25834fa03afb0; no voters:
I20260810 23:04:05.004271 7261 raft_consensus.cc:2781] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:04:05.004372 7261 raft_consensus.cc:482] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:04:05.004424 7261 raft_consensus.cc:3037] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:05.005560 7261 raft_consensus.cc:504] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:05.005966 7261 leader_election.cc:288] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [CANDIDATE]: Term 1 election: Requested vote from peers 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117), 71725911b2a44f989f68a583ba8a0004 (127.15.56.1:40503)
I20260810 23:04:05.006413 7159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ae0343e6054eb99538e62f228b781b" candidate_uuid: "16f5678be0b141698d45e5d6a5c2ae77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8f7162548d4c488ec25834fa03afb0"
I20260810 23:04:05.006534 6759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ae0343e6054eb99538e62f228b781b" candidate_uuid: "16f5678be0b141698d45e5d6a5c2ae77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71725911b2a44f989f68a583ba8a0004"
I20260810 23:04:05.006577 7159 raft_consensus.cc:3037] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:05.006686 6759 raft_consensus.cc:3037] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:05.007807 6759 raft_consensus.cc:2454] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5678be0b141698d45e5d6a5c2ae77 in term 1.
I20260810 23:04:05.007808 7159 raft_consensus.cc:2454] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5678be0b141698d45e5d6a5c2ae77 in term 1.
I20260810 23:04:05.008152 6859 leader_election.cc:302] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5678be0b141698d45e5d6a5c2ae77, 71725911b2a44f989f68a583ba8a0004; no voters:
I20260810 23:04:05.008350 7261 raft_consensus.cc:2781] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:05.008597 7261 raft_consensus.cc:686] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 1 LEADER]: Becoming Leader. State: Replica: 16f5678be0b141698d45e5d6a5c2ae77, State: Running, Role: LEADER
I20260810 23:04:05.008956 7261 consensus_queue.cc:237] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:05.012498 6545 catalog_manager.cc:5495] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2). New cstate: current_term: 1 leader_uuid: "16f5678be0b141698d45e5d6a5c2ae77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } health_report { overall_health: UNKNOWN } } }
I20260810 23:04:05.044611 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:05.040834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.15.56.2:37171)
0810 23:04:05.040973 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:05.040977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:05.040991 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:04:05.041101 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:05.041104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:05.041167 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:05.041260 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:04:05.041482 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.042009 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.042918 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.043180 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.043266 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:05.043620 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:04:05.044113 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:04:05.044255 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:05.044370 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 23:04:05.044429 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:04:05.051116 6759 raft_consensus.cc:1261] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Refusing update from remote peer 16f5678be0b141698d45e5d6a5c2ae77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:05.051122 7159 raft_consensus.cc:1261] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0 [term 1 FOLLOWER]: Refusing update from remote peer 16f5678be0b141698d45e5d6a5c2ae77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:05.051893 7261 consensus_queue.cc:1035] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:05.052114 7264 consensus_queue.cc:1035] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:05.062729 7271 mvcc.cc:205] Tried to move back new op lower bound from 7317106872521109504 to 7317106872360108032. Current Snapshot: MvccSnapshot[applied={T|T < 7317106872521109504}]
I20260810 23:04:05.068917 6547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:05.069137 6547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:05.072028 6547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Sending DeleteTablet for 3 replicas of tablet c6ae0343e6054eb99538e62f228b781b
I20260810 23:04:05.072916 6724 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ae0343e6054eb99538e62f228b781b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:05 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45164
I20260810 23:04:05.072947 7130 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ae0343e6054eb99538e62f228b781b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:05 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45982
I20260810 23:04:05.072937 6925 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ae0343e6054eb99538e62f228b781b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:05 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:53836
I20260810 23:04:05.073547 7282 tablet_replica.cc:330] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: stopping tablet replica
I20260810 23:04:05.073694 7282 raft_consensus.cc:2229] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:05.073807 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 6607
I20260810 23:04:05.073856 7282 raft_consensus.cc:2258] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:05.105247 7285 tablet_replica.cc:330] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77: stopping tablet replica
I20260810 23:04:05.105407 7284 tablet_replica.cc:330] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0: stopping tablet replica
I20260810 23:04:05.105875 7284 raft_consensus.cc:2229] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:05.105875 7285 raft_consensus.cc:2229] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:04:05.106307 7284 raft_consensus.cc:2258] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:05.106498 7285 raft_consensus.cc:2258] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:05.108172 7284 ts_tablet_manager.cc:1905] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:05.108696 7285 ts_tablet_manager.cc:1905] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:05.112128 7285 ts_tablet_manager.cc:1918] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:05.112349 7284 ts_tablet_manager.cc:1918] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:05.112393 7285 log.cc:1196] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/c6ae0343e6054eb99538e62f228b781b
I20260810 23:04:05.112516 7284 log.cc:1196] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/c6ae0343e6054eb99538e62f228b781b
I20260810 23:04:05.112912 7284 ts_tablet_manager.cc:1939] T c6ae0343e6054eb99538e62f228b781b P 7e8f7162548d4c488ec25834fa03afb0: Deleting consensus metadata
I20260810 23:04:05.112941 7285 ts_tablet_manager.cc:1939] T c6ae0343e6054eb99538e62f228b781b P 16f5678be0b141698d45e5d6a5c2ae77: Deleting consensus metadata
I20260810 23:04:05.114681 6530 catalog_manager.cc:4841] TS 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171): tablet c6ae0343e6054eb99538e62f228b781b (table test-table [id=8cc4ab9503134f19938713b46476f43a]) successfully deleted
W20260810 23:04:05.114738 6532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:40503 (error 108)
I20260810 23:04:05.114905 6528 catalog_manager.cc:4841] TS 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117): tablet c6ae0343e6054eb99538e62f228b781b (table test-table [id=8cc4ab9503134f19938713b46476f43a]) successfully deleted
I20260810 23:04:05.115936 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:05.115577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:40503 (local address 127.0.0.1:45178)
0810 23:04:05.115741 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:05.115825 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:04:05.116958 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 6822
W20260810 23:04:05.116793 6532 catalog_manager.cc:4584] TS 71725911b2a44f989f68a583ba8a0004 (127.15.56.1:40503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6ae0343e6054eb99538e62f228b781b: Network error: Client connection negotiation failed: client connection to 127.15.56.1:40503: connect: Connection refused (error 111)
I20260810 23:04:05.143622 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 7026
I20260810 23:04:05.168303 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 6507
I20260810 23:04:05.182291 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44547
--webserver_interface=127.15.56.62
--webserver_port=32949
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:44547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:05.306083 7287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:05.306437 7287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:05.306501 7287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:05.311019 7287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:04:05.311124 7287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:05.311159 7287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:05.311182 7287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:04:05.311203 7287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:04:05.317044 7287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:44547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:05.318331 7287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:05.319495 7287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:05.329154 7297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:05.329213 7287 server_base.cc:1034] running on GCE node
W20260810 23:04:05.329159 7293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:05.329154 7294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:05.330044 7287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:05.331305 7287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:05.332515 7287 hybrid_clock.cc:648] HybridClock initialized: now 1786403045332470 us; error 57 us; skew 500 ppm
Aug 10 23:04:05 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403045, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:05.335858 7287 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:04:05.337021 7287 webserver.cc:460] Webserver started at http://127.15.56.62:32949/ using document root <none> and password file <none>
I20260810 23:04:05.337651 7287 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:05.337728 7287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:05.342322 7287 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:04:05.344759 7309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:05.346132 7287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:04:05.346266 7287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "c8ffc07682a24ae2b5da1d1b1db3003e"
format_stamp: "Formatted at 2026-08-10 23:04:02 on dist-test-slave-6bbx"
server_key: "7124602168ea99fce4d2af84347c31a4"
server_key_iv: "3bbfc056c10a7a80d7f2467920425f26"
server_key_version: "encryptionkey@0"
I20260810 23:04:05.346872 7287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:05.371802 7287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:05.374949 7287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:05.375154 7287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:05.382627 7287 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:44547
I20260810 23:04:05.382912 7389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:44547 every 8 connection(s)
I20260810 23:04:05.383885 7287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:04:05.388713 7390 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:04:05.390604 7390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Bootstrap starting.
I20260810 23:04:05.391006 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 7287
I20260810 23:04:05.391465 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:40503
--local_ip_for_outbound_sockets=127.15.56.1
--tserver_master_addrs=127.15.56.62:44547
--webserver_port=41723
--webserver_interface=127.15.56.1
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:04:05.410553 7390 log.cc:824] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:05.423390 7390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:04:05.423864 7390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e: Bootstrap complete.
I20260810 23:04:05.428200 7390 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:05.428632 7390 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ffc07682a24ae2b5da1d1b1db3003e, State: Initialized, Role: FOLLOWER
I20260810 23:04:05.429180 7390 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:05.429317 7390 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:04:05.429386 7390 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:04:05.429489 7390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:04:05.431555 7390 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:05.431931 7390 leader_election.cc:302] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8ffc07682a24ae2b5da1d1b1db3003e; no voters:
I20260810 23:04:05.432266 7390 leader_election.cc:288] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:04:05.432379 7394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:04:05.432564 7394 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [term 3 LEADER]: Becoming Leader. State: Replica: c8ffc07682a24ae2b5da1d1b1db3003e, State: Running, Role: LEADER
I20260810 23:04:05.432888 7394 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } }
I20260810 23:04:05.433606 7390 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:04:05.433921 7396 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8ffc07682a24ae2b5da1d1b1db3003e. Latest consensus state: current_term: 3 leader_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } } }
I20260810 23:04:05.434057 7396 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:05.434293 7395 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ffc07682a24ae2b5da1d1b1db3003e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 44547 } } }
I20260810 23:04:05.434404 7395 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ffc07682a24ae2b5da1d1b1db3003e [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:05.434731 7399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:04:05.440534 7399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:04:05.441493 7399 catalog_manager.cc:1259] Loaded cluster ID: ac64f1c9168e437c9fe7a20143006f40
I20260810 23:04:05.441562 7399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 23:04:05.442833 7399 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb6d1e4d9ad google::LogMessage::Fail() at ??:0
@ 0x7fb6d1e5088a google::LogMessage::SendToLog() at ??:0
@ 0x7fb6d1e4d4de google::LogMessage::Flush() at ??:0
@ 0x7fb6d1e4ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb6dc78cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb6dc78dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb6dc78a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb6dc7ce640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb6d409f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb6d409fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb6d40a13f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb6d4090512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb6d56d46db start_thread at ??:0
@ 0x7fb6d094b61f clone at ??:0
W20260810 23:04:05.545527 7392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:05.545818 7392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:05.545863 7392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:05.549996 7392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:05.550071 7392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:05.550134 7392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:04:05.555768 7392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:40503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=41723
--tserver_master_addrs=127.15.56.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:05.556969 7392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:05.558094 7392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:05.568938 7452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:05.568924 7453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:05.569443 7392 server_base.cc:1034] running on GCE node
W20260810 23:04:05.568924 7455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:05.570034 7392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:05.570757 7392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:05.571939 7392 hybrid_clock.cc:648] HybridClock initialized: now 1786403045571909 us; error 41 us; skew 500 ppm
Aug 10 23:04:05 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403045, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:05.575156 7392 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:04:05.576311 7392 webserver.cc:460] Webserver started at http://127.15.56.1:41723/ using document root <none> and password file <none>
I20260810 23:04:05.576937 7392 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:05.577013 7392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:05.581794 7392 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:04:05.584548 7464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:05.586243 7392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:04:05.586421 7392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "71725911b2a44f989f68a583ba8a0004"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "06bdc983af5ce1a6ff10ab211bdfeccd"
server_key_iv: "27b6798194683e42142149919290c741"
server_key_version: "encryptionkey@0"
I20260810 23:04:05.587065 7392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:05.611243 7392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:05.614972 7392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:05.615164 7392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:05.615837 7392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:05.617080 7475 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:04:05.625779 7392 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:04:05.625888 7392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.001s
I20260810 23:04:05.625967 7392 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:04:05.627952 7392 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:04:05.628014 7392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260810 23:04:05.628362 7475 tablet_bootstrap.cc:492] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: Bootstrap starting.
I20260810 23:04:05.638345 7392 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:40503
I20260810 23:04:05.638635 7631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:40503 every 8 connection(s)
I20260810 23:04:05.639418 7392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:04:05.640969 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 7392
I20260810 23:04:05.641350 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:37171
--local_ip_for_outbound_sockets=127.15.56.2
--tserver_master_addrs=127.15.56.62:44547
--webserver_port=41759
--webserver_interface=127.15.56.2
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:04:05.650954 7475 log.cc:824] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:05.658849 7475 tablet_bootstrap.cc:492] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:04:05.659255 7475 tablet_bootstrap.cc:492] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: Bootstrap complete.
I20260810 23:04:05.660512 7475 ts_tablet_manager.cc:1397] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
W20260810 23:04:05.663424 7636 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:05.641645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:49023)
0810 23:04:05.642918 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:05.642978 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:05.644073 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:05.662354 (+ 18281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":150,"threads_started":1}
I20260810 23:04:05.663467 7475 raft_consensus.cc:348] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:05.664026 7475 raft_consensus.cc:729] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71725911b2a44f989f68a583ba8a0004, State: Initialized, Role: FOLLOWER
W20260810 23:04:05.664077 7634 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:44547 (0 consecutive failures): Network error: Failed to ping master at 127.15.56.62:44547: Client connection negotiation failed: client connection to 127.15.56.62:44547: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:04:05.664582 7475 consensus_queue.cc:260] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:05.665019 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:05.664687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:42731)
0810 23:04:05.664871 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:05.664942 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 23:04:05.665675 7475 ts_tablet_manager.cc:1428] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260810 23:04:05.665814 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:05.665547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:40159)
0810 23:04:05.665683 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:05.665741 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 23:04:05.666016 7634 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:04:05.666522 7636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:05.666326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:34193)
0810 23:04:05.666428 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:05.666471 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 23:04:05.768954 7637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:05.769240 7637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:05.769287 7637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:05.773362 7637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:05.773434 7637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:05.773495 7637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:04:05.779117 7637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:37171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=41759
--tserver_master_addrs=127.15.56.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:05.780377 7637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:05.781419 7637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:05.791931 7647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:05.791904 7644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:05.792094 7637 server_base.cc:1034] running on GCE node
W20260810 23:04:05.791893 7645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:05.792550 7637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:05.793221 7637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:05.794405 7637 hybrid_clock.cc:648] HybridClock initialized: now 1786403045794380 us; error 36 us; skew 500 ppm
Aug 10 23:04:05 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403045, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:05.797256 7637 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:04:05.798207 7637 webserver.cc:460] Webserver started at http://127.15.56.2:41759/ using document root <none> and password file <none>
I20260810 23:04:05.798789 7637 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:05.798861 7637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:05.802963 7637 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:04:05.805123 7661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:05.806416 7637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:04:05.806529 7637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "16f5678be0b141698d45e5d6a5c2ae77"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "5c9b625989ff1c1f1101cd821140426c"
server_key_iv: "ec406d663dfb4c9ddfe5c4026d7bf0ea"
server_key_version: "encryptionkey@0"
I20260810 23:04:05.807083 7637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:05.836817 7637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:05.840250 7637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:05.840459 7637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:05.841162 7637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:05.842062 7637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:05.842126 7637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:05.842187 7637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:05.842212 7637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:05.851161 7637 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:37171
I20260810 23:04:05.851238 7822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:37171 every 8 connection(s)
I20260810 23:04:05.852241 7637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:04:05.856352 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:05.854472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:45071)
0810 23:04:05.854965 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:05.855085 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":120,"threads_started":1}
W20260810 23:04:05.856828 7823 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:44547 (0 consecutive failures): Network error: Failed to ping master at 127.15.56.62:44547: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
I20260810 23:04:05.857456 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:05.857210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:36715)
0810 23:04:05.857347 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:05.857399 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 23:04:05.858068 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:05.857872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:59027)
0810 23:04:05.857972 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:05.858018 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:04:05.858245 7823 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:04:05.861339 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 7637
I20260810 23:04:05.861754 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:43117
--local_ip_for_outbound_sockets=127.15.56.3
--tserver_master_addrs=127.15.56.62:44547
--webserver_port=34287
--webserver_interface=127.15.56.3
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:37233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:05.980907 7826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:05.981189 7826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:05.981235 7826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:05.985285 7826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:05.985363 7826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:05.985427 7826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:04:05.991158 7826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:37233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:43117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=34287
--tserver_master_addrs=127.15.56.62:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:05.992477 7826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:05.993588 7826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:06.003716 7836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:06.003746 7835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:06.003808 7826 server_base.cc:1034] running on GCE node
W20260810 23:04:06.003703 7839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:06.004386 7826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:06.005105 7826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:06.006266 7826 hybrid_clock.cc:648] HybridClock initialized: now 1786403046006248 us; error 28 us; skew 500 ppm
Aug 10 23:04:06 dist-test-slave-6bbx krb5kdc[5717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:06.009064 7826 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:04:06.009968 7826 webserver.cc:460] Webserver started at http://127.15.56.3:34287/ using document root <none> and password file <none>
I20260810 23:04:06.010571 7826 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:06.010643 7826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:06.014624 7826 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:04:06.016885 7851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.018191 7826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 23:04:06.018313 7826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "7e8f7162548d4c488ec25834fa03afb0"
format_stamp: "Formatted at 2026-08-10 23:04:03 on dist-test-slave-6bbx"
server_key: "0e57d9edffaeb261477e39f4df86fc78"
server_key_iv: "020d1c27e7b1b6d0148dabf90597c443"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.018926 7826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:06.031989 7826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:06.035197 7826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:06.035409 7826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:06.036139 7826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:06.037111 7826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:06.037165 7826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.037223 7826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:06.037253 7826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.046574 7826 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:43117
I20260810 23:04:06.046657 8018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:43117 every 8 connection(s)
I20260810 23:04:06.047760 7826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:04:06.050164 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 7826
I20260810 23:04:06.052107 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.050048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:55567)
0810 23:04:06.050575 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:06.050727 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":119,"threads_started":1}
W20260810 23:04:06.052574 8020 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:44547 (0 consecutive failures): Network error: Failed to ping master at 127.15.56.62:44547: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
I20260810 23:04:06.053220 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.052985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:42503)
0810 23:04:06.053116 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:06.053162 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:04:06.053835 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.053622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:32991)
0810 23:04:06.053724 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:06.053783 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:04:06.053990 8020 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:04:06.667752 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.667227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:33659)
0810 23:04:06.667527 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:06.667593 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":78,"threads_started":1}
I20260810 23:04:06.820310 8028 raft_consensus.cc:482] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:04:06.820542 8028 raft_consensus.cc:504] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:06.821694 8028 leader_election.cc:288] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171), 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117)
Aug 10 23:04:06 dist-test-slave-6bbx krb5kdc[5717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403045, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.3@KRBTEST.COM
Aug 10 23:04:06 dist-test-slave-6bbx krb5kdc[5717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403045, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.2@KRBTEST.COM
I20260810 23:04:06.838300 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.821984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:43117 (local address 127.15.56.1:48835)
0810 23:04:06.822114 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:06.822129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:06.822240 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:06.824913 (+ 2673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:06.824918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:06.825306 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:06.825820 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.825836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.827727 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.827730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.828693 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.828704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.829421 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.829424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.829610 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.830282 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:06.830294 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:06.831973 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:06.834636 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:06.834644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:06.834663 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:06.835210 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:06.835756 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:06.835760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:06.835762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:06.836035 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:06.837008 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:06.837016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:06.837429 (+ 413us) client_negotiation.cc:769] Sending connection context
0810 23:04:06.837671 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 23:04:06.838073 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":110,"spinlock_wait_cycles":25600}
I20260810 23:04:06.838392 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.822156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:43699 (local address 127.15.56.2:37171)
0810 23:04:06.822483 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:06.822499 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:06.822937 (+ 438us) server_negotiation.cc:407] Connection header received
0810 23:04:06.823982 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:06.823988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:06.824473 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:06.824815 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:06.826384 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.827823 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.828811 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.829288 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.829440 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.832468 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:06.832473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:06.832485 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:06.832516 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:06.834471 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:06.835378 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:06.835386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:06.835393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:06.835601 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:06.836177 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:06.836181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:06.836183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:06.836849 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:06.836995 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:06.837831 (+ 836us) server_negotiation.cc:299] Negotiation successful
0810 23:04:06.838176 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":118,"threads_started":1}
I20260810 23:04:06.838383 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.822438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:48835 (local address 127.15.56.3:43117)
0810 23:04:06.823137 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:06.823151 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:06.823191 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:04:06.823971 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:06.823981 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:06.824473 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:06.824815 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:06.825996 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.827539 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.828832 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.829297 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.829449 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.832257 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:06.832269 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:06.832284 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:06.832320 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:06.834471 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:06.835380 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:06.835386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:06.835393 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:06.835601 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:06.836177 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:06.836181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:06.836183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:06.836854 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:06.836995 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:06.837838 (+ 843us) server_negotiation.cc:299] Negotiation successful
0810 23:04:06.838176 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":83,"threads_started":1}
I20260810 23:04:06.838299 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.821984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:37171 (local address 127.15.56.1:43699)
0810 23:04:06.822809 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:06.822823 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:06.822937 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:06.825973 (+ 3036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:06.825975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:06.826138 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:06.826231 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.826236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.827956 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.827959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.828689 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.828698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.829409 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.829413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.829639 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.830230 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:06.830249 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:06.832276 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:06.834636 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:06.834644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:06.834663 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:06.835210 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:06.835761 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:06.835763 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:06.835765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:06.836035 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:06.837008 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:06.837016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:06.837429 (+ 413us) client_negotiation.cc:769] Sending connection context
0810 23:04:06.837671 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 23:04:06.838144 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":384,"spinlock_wait_cycles":44672,"thread_start_us":68,"threads_started":1}
W20260810 23:04:06.841749 7469 leader_election.cc:341] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171): Not found: Tablet not found: c6ae0343e6054eb99538e62f228b781b
W20260810 23:04:06.842043 7467 leader_election.cc:341] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117): Not found: Tablet not found: c6ae0343e6054eb99538e62f228b781b
I20260810 23:04:06.842128 7467 leader_election.cc:302] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71725911b2a44f989f68a583ba8a0004; no voters: 16f5678be0b141698d45e5d6a5c2ae77, 7e8f7162548d4c488ec25834fa03afb0
I20260810 23:04:06.842403 8028 raft_consensus.cc:2726] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:04:06.859313 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.858863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:58177)
0810 23:04:06.859133 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:06.859195 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260810 23:04:07.055161 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:07.054691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:36673)
0810 23:04:07.054988 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:07.055045 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":102,"threads_started":1}
I20260810 23:04:07.669139 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:07.668618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:39649)
0810 23:04:07.668935 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:07.669001 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
I20260810 23:04:07.860606 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:07.860132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:45431)
0810 23:04:07.860420 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:07.860484 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260810 23:04:08.056406 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:08.055958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:59911)
0810 23:04:08.056221 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:08.056282 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260810 23:04:08.496256 8041 raft_consensus.cc:482] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:04:08.496390 8041 raft_consensus.cc:504] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:08.496765 8041 leader_election.cc:288] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171), 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117)
W20260810 23:04:08.497584 7467 leader_election.cc:341] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117): Not found: Tablet not found: c6ae0343e6054eb99538e62f228b781b
W20260810 23:04:08.497695 7469 leader_election.cc:341] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171): Not found: Tablet not found: c6ae0343e6054eb99538e62f228b781b
I20260810 23:04:08.497804 7469 leader_election.cc:302] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71725911b2a44f989f68a583ba8a0004; no voters: 16f5678be0b141698d45e5d6a5c2ae77, 7e8f7162548d4c488ec25834fa03afb0
I20260810 23:04:08.497959 8041 raft_consensus.cc:2726] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:04:08.670531 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:08.670056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:41819)
0810 23:04:08.670362 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:08.670428 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":81,"threads_started":1}
I20260810 23:04:08.861831 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:08.861354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:54989)
0810 23:04:08.861643 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:08.861724 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260810 23:04:09.057636 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.057227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:60461)
0810 23:04:09.057483 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:09.057537 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260810 23:04:09.671830 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.671296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:47049)
0810 23:04:09.671596 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:09.671707 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":75,"threads_started":1}
I20260810 23:04:09.863296 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.862792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:43795)
0810 23:04:09.863108 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:09.863187 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":89,"threads_started":1}
I20260810 23:04:10.058890 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:10.058450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:53325)
0810 23:04:10.058725 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:10.058789 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260810 23:04:10.673311 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:10.672796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:34465)
0810 23:04:10.673129 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:10.673198 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":83,"threads_started":1}
I20260810 23:04:10.864812 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:10.864337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:56991)
0810 23:04:10.864643 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:10.864705 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":86,"threads_started":1}
I20260810 23:04:10.983924 8056 raft_consensus.cc:482] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:04:10.984087 8056 raft_consensus.cc:504] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:10.984546 8056 leader_election.cc:288] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171), 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117)
W20260810 23:04:10.985422 7469 leader_election.cc:341] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171): Not found: Tablet not found: c6ae0343e6054eb99538e62f228b781b
W20260810 23:04:10.985549 7467 leader_election.cc:341] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117): Not found: Tablet not found: c6ae0343e6054eb99538e62f228b781b
I20260810 23:04:10.985641 7467 leader_election.cc:302] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71725911b2a44f989f68a583ba8a0004; no voters: 16f5678be0b141698d45e5d6a5c2ae77, 7e8f7162548d4c488ec25834fa03afb0
I20260810 23:04:10.985808 8056 raft_consensus.cc:2726] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:04:11.060309 8057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:11.059840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:33231)
0810 23:04:11.060138 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:11.060204 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
I20260810 23:04:11.674834 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:11.674246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:58699)
0810 23:04:11.674604 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:11.674675 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":104,"threads_started":1}
I20260810 23:04:11.866079 8061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:11.865619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:46473)
0810 23:04:11.865916 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:11.865978 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
I20260810 23:04:12.062296 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:12.061446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:38271)
0810 23:04:12.061966 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:12.062086 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":159,"threads_started":1}
I20260810 23:04:12.676162 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:12.675644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:57133)
0810 23:04:12.675984 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:12.676053 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
I20260810 23:04:12.868099 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:12.866992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:48949)
0810 23:04:12.867398 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:12.867570 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":91,"threads_started":1}
I20260810 23:04:13.064390 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:13.063566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:48469)
0810 23:04:13.064110 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:13.064213 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":127,"threads_started":1}
I20260810 23:04:13.677388 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:13.676902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:58329)
0810 23:04:13.677218 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:13.677280 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":85,"threads_started":1}
I20260810 23:04:13.869637 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:13.869063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:45685)
0810 23:04:13.869450 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:13.869522 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":126,"threads_started":1}
I20260810 23:04:14.065760 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:14.065295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:53759)
0810 23:04:14.065583 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:14.065650 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
I20260810 23:04:14.678680 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:14.678191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:37841)
0810 23:04:14.678495 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:14.678557 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
I20260810 23:04:14.844962 8072 raft_consensus.cc:482] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:04:14.845131 8072 raft_consensus.cc:504] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:14.845722 8072 leader_election.cc:288] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171), 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117)
W20260810 23:04:14.846639 7467 leader_election.cc:341] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117): Not found: Tablet not found: c6ae0343e6054eb99538e62f228b781b
W20260810 23:04:14.846798 7469 leader_election.cc:341] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171): Not found: Tablet not found: c6ae0343e6054eb99538e62f228b781b
I20260810 23:04:14.846872 7469 leader_election.cc:302] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71725911b2a44f989f68a583ba8a0004; no voters: 16f5678be0b141698d45e5d6a5c2ae77, 7e8f7162548d4c488ec25834fa03afb0
I20260810 23:04:14.847034 8072 raft_consensus.cc:2726] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:04:14.870996 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:14.870539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:46243)
0810 23:04:14.870824 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:14.870889 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260810 23:04:15.067157 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:15.066664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:60607)
0810 23:04:15.066977 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:15.067047 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":82,"threads_started":1}
I20260810 23:04:15.680140 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:15.679613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:47945)
0810 23:04:15.679961 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:15.680026 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260810 23:04:15.872330 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:15.871835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:57165)
0810 23:04:15.872156 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:15.872224 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":78,"threads_started":1}
I20260810 23:04:16.068603 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:16.068083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:60051)
0810 23:04:16.068424 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:16.068492 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":98,"threads_started":1}
I20260810 23:04:16.681504 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:16.681006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:60155)
0810 23:04:16.681318 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:16.681391 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":89,"threads_started":1}
I20260810 23:04:16.873713 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:16.873222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:52655)
0810 23:04:16.873536 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:16.873603 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":85,"threads_started":1}
I20260810 23:04:17.069945 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:17.069478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:59083)
0810 23:04:17.069771 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:17.069836 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260810 23:04:17.682989 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:17.682485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:56153)
0810 23:04:17.682809 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:17.682878 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260810 23:04:17.875026 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:17.874570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:36271)
0810 23:04:17.874864 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:17.874921 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
I20260810 23:04:18.071385 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:18.070796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:56339)
0810 23:04:18.071198 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:18.071271 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260810 23:04:18.684536 8089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:18.684049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:40397)
0810 23:04:18.684368 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:18.684432 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
I20260810 23:04:18.816516 8090 raft_consensus.cc:482] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:04:18.816658 8090 raft_consensus.cc:504] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5678be0b141698d45e5d6a5c2ae77" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 37171 } } peers { permanent_uuid: "7e8f7162548d4c488ec25834fa03afb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43117 } } peers { permanent_uuid: "71725911b2a44f989f68a583ba8a0004" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 40503 } }
I20260810 23:04:18.817128 8090 leader_election.cc:288] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171), 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117)
W20260810 23:04:18.817912 7467 leader_election.cc:341] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e8f7162548d4c488ec25834fa03afb0 (127.15.56.3:43117): Not found: Tablet not found: c6ae0343e6054eb99538e62f228b781b
W20260810 23:04:18.818013 7469 leader_election.cc:341] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16f5678be0b141698d45e5d6a5c2ae77 (127.15.56.2:37171): Not found: Tablet not found: c6ae0343e6054eb99538e62f228b781b
I20260810 23:04:18.818122 7469 leader_election.cc:302] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 71725911b2a44f989f68a583ba8a0004; no voters: 16f5678be0b141698d45e5d6a5c2ae77, 7e8f7162548d4c488ec25834fa03afb0
I20260810 23:04:18.818279 8090 raft_consensus.cc:2726] T c6ae0343e6054eb99538e62f228b781b P 71725911b2a44f989f68a583ba8a0004 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:04:18.876372 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:18.875881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:60457)
0810 23:04:18.876197 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:18.876265 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260810 23:04:19.072587 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:19.072138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:33817)
0810 23:04:19.072420 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:19.072481 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260810 23:04:19.685925 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:19.685441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:53369)
0810 23:04:19.685717 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:19.685787 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260810 23:04:19.877681 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:19.877245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:54793)
0810 23:04:19.877511 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:19.877574 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
I20260810 23:04:20.073834 8097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:20.073394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:50689)
0810 23:04:20.073666 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:20.073725 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260810 23:04:20.687788 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:20.687166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.1:55259)
0810 23:04:20.687554 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:20.687642 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":99,"threads_started":1}
I20260810 23:04:20.879145 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:20.878684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.2:48155)
0810 23:04:20.878983 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:20.879041 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":93,"threads_started":1}
I20260810 23:04:21.052207 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 7392
I20260810 23:04:21.065313 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 7637
I20260810 23:04:21.075503 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:21.074903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:44547 (local address 127.15.56.3:38379)
0810 23:04:21.075299 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:21.075369 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":80,"threads_started":1}
I20260810 23:04:21.081499 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 7826
2026-08-10T23:04:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19664 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:04:21 dist-test-slave-6bbx krb5kdc[8105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:04:21 dist-test-slave-6bbx krb5kdc[8105](info): set up 2 sockets
Aug 10 23:04:21 dist-test-slave-6bbx krb5kdc[8105](info): commencing operation
krb5kdc: starting...
W20260810 23:04:22.255154 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:04:22 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:04:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:04:22.384536 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33657
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:33657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:22.507148 8120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:22.507498 8120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:22.507560 8120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:22.511811 8120 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:04:22.511881 8120 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:04:22.511901 8120 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:04:22.511920 8120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:04:22.511945 8120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:22.511968 8120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:22.511999 8120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:04:22.512022 8120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:04:22.517706 8120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:33657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:22.518898 8120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:22.520006 8120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:22.528959 8128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:22.528976 8129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:22.529026 8120 server_base.cc:1034] running on GCE node
W20260810 23:04:22.528954 8131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:22.529731 8120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:22.530886 8120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:22.532040 8120 hybrid_clock.cc:648] HybridClock initialized: now 1786403062532025 us; error 33 us; skew 500 ppm
Aug 10 23:04:22 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403062, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:22.534922 8120 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:04:22.536010 8120 webserver.cc:460] Webserver started at http://127.15.56.62:45279/ using document root <none> and password file <none>
I20260810 23:04:22.536630 8120 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:22.536708 8120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:22.536937 8120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:22.539237 8120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "123e3d443f6946b4a052db96b8313ee9"
format_stamp: "Formatted at 2026-08-10 23:04:22 on dist-test-slave-6bbx"
server_key: "a6296df0d28b5953ffb8900d4db4bc42"
server_key_iv: "70b977ff14b6611bdce4f4a8dbdf9507"
server_key_version: "encryptionkey@0"
I20260810 23:04:22.539865 8120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "123e3d443f6946b4a052db96b8313ee9"
format_stamp: "Formatted at 2026-08-10 23:04:22 on dist-test-slave-6bbx"
server_key: "a6296df0d28b5953ffb8900d4db4bc42"
server_key_iv: "70b977ff14b6611bdce4f4a8dbdf9507"
server_key_version: "encryptionkey@0"
I20260810 23:04:22.544173 8120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:04:22.546923 8140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:22.548324 8120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:04:22.548460 8120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "123e3d443f6946b4a052db96b8313ee9"
format_stamp: "Formatted at 2026-08-10 23:04:22 on dist-test-slave-6bbx"
server_key: "a6296df0d28b5953ffb8900d4db4bc42"
server_key_iv: "70b977ff14b6611bdce4f4a8dbdf9507"
server_key_version: "encryptionkey@0"
I20260810 23:04:22.548569 8120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:22.575245 8120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:22.578678 8120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:22.578873 8120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:22.586819 8120 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:33657
I20260810 23:04:22.587090 8215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:33657 every 8 connection(s)
I20260810 23:04:22.588003 8120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:04:22.591465 8216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:22.592711 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 8120
I20260810 23:04:22.592825 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:04:22.593371 15584 external_mini_cluster.cc:1428] Setting key 8c0347daf8a17379d592ba27679e9668
I20260810 23:04:22.599146 8216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Bootstrap starting.
I20260810 23:04:22.602224 8216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:22.603055 8216 log.cc:824] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:04:22 dist-test-slave-6bbx krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:22.605753 8216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: No bootstrap required, opened a new log
I20260810 23:04:22.610452 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:22.595056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.15.56.62:33657)
0810 23:04:22.595587 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:22.595608 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:22.595656 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:04:22.596634 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:22.596661 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:22.597150 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:22.597565 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:22.598360 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:22.599851 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:22.600909 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:22.601633 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:22.601836 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:22.604580 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:22.604587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:22.604606 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:22.604657 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:22.606850 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:22.607383 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:22.607390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:22.607398 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:22.607551 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:22.607925 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:22.607929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:22.607931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:22.608531 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:22.608748 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:22.609041 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 23:04:22.609381 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":171,"threads_started":1}
I20260810 23:04:22.610625 8216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:04:22.610908 8216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:22.610973 8216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123e3d443f6946b4a052db96b8313ee9, State: Initialized, Role: FOLLOWER
I20260810 23:04:22.611441 8216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:04:22.611572 8216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:04:22.611635 8216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:04:22.611792 8216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:22.612771 8216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:04:22.613121 8216 leader_election.cc:302] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 123e3d443f6946b4a052db96b8313ee9; no voters:
I20260810 23:04:22.613449 8216 leader_election.cc:288] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:04:22.613545 8222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:22.613775 8222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 1 LEADER]: Becoming Leader. State: Replica: 123e3d443f6946b4a052db96b8313ee9, State: Running, Role: LEADER
I20260810 23:04:22.614128 8222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:04:22.614645 8216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:04:22.615741 8224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 123e3d443f6946b4a052db96b8313ee9. Latest consensus state: current_term: 1 leader_uuid: "123e3d443f6946b4a052db96b8313ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } } }
I20260810 23:04:22.615880 8224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:22.616359 8223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "123e3d443f6946b4a052db96b8313ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } } }
I20260810 23:04:22.616487 8223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: This master's current role is: LEADER
W20260810 23:04:22.619612 8245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:04:22.619748 8245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:04:22.619967 8239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:04:22.620939 8239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:04:22.626868 8239 catalog_manager.cc:1347] Generated new cluster ID: 296558d7dbb7434d854158f5673db1ba
I20260810 23:04:22.626956 8239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:04:22.653613 8239 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:04:22.654958 8239 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:04:22.665203 8239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Generated new TSK 0
I20260810 23:04:22.666174 8239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:04:22.701018 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33657
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:22.823513 8251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:22.823866 8251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:22.823920 8251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:22.828053 8251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:22.828118 8251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:22.828194 8251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:04:22.833702 8251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:22.834825 8251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:22.835932 8251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:22.846496 8263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:22.846498 8266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:22.846498 8262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:22.846890 8251 server_base.cc:1034] running on GCE node
I20260810 23:04:22.847357 8251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:22.848186 8251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:22.849390 8251 hybrid_clock.cc:648] HybridClock initialized: now 1786403062849365 us; error 34 us; skew 500 ppm
Aug 10 23:04:22 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403062, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:22.853092 8251 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:04:22.854238 8251 webserver.cc:460] Webserver started at http://127.15.56.1:45271/ using document root <none> and password file <none>
I20260810 23:04:22.854859 8251 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:22.854929 8251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:22.855142 8251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:22.857522 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "5c4239483857493fad13d728aa0e9cdd"
format_stamp: "Formatted at 2026-08-10 23:04:22 on dist-test-slave-6bbx"
server_key: "4b6a7c027b1283fcbd312401b15739a8"
server_key_iv: "fa68cfac543e83c6102bf9b41a3bc44e"
server_key_version: "encryptionkey@0"
I20260810 23:04:22.858140 8251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "5c4239483857493fad13d728aa0e9cdd"
format_stamp: "Formatted at 2026-08-10 23:04:22 on dist-test-slave-6bbx"
server_key: "4b6a7c027b1283fcbd312401b15739a8"
server_key_iv: "fa68cfac543e83c6102bf9b41a3bc44e"
server_key_version: "encryptionkey@0"
I20260810 23:04:22.862528 8251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:04:22.865636 8274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:22.867067 8251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:04:22.867220 8251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "5c4239483857493fad13d728aa0e9cdd"
format_stamp: "Formatted at 2026-08-10 23:04:22 on dist-test-slave-6bbx"
server_key: "4b6a7c027b1283fcbd312401b15739a8"
server_key_iv: "fa68cfac543e83c6102bf9b41a3bc44e"
server_key_version: "encryptionkey@0"
I20260810 23:04:22.867343 8251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:22.878530 8251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:22.881933 8251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:22.882141 8251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:22.882755 8251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:22.883617 8251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:22.883702 8251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:22.883767 8251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:22.883795 8251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:22.893101 8251 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:35907
I20260810 23:04:22.893193 8446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:35907 every 8 connection(s)
I20260810 23:04:22.894084 8251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:04:22.898929 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 8251
I20260810 23:04:22.899072 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:04:22.899726 15584 external_mini_cluster.cc:1428] Setting key 614056285138a9d6971b0e2b9b7d1382
Aug 10 23:04:22 dist-test-slave-6bbx krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403062, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:22.909669 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:22.896042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:35821 (local address 127.15.56.62:33657)
0810 23:04:22.896172 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:22.896176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:22.897111 (+ 935us) server_negotiation.cc:407] Connection header received
0810 23:04:22.898576 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:22.898579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:22.898634 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:22.898711 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:22.900592 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:22.901301 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:22.902571 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:22.902897 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:22.903007 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:22.905677 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:22.905680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:22.905682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:22.905709 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:22.907267 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:22.907893 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:22.907897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:22.907899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:22.908006 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:22.908382 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:22.908385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:22.908387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:22.908708 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:22.908817 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:22.909337 (+ 520us) server_negotiation.cc:299] Negotiation successful
0810 23:04:22.909488 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:04:22.910487 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:22.896443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:35821)
0810 23:04:22.896939 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:22.896987 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:22.898339 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:22.898951 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:22.898962 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:22.899720 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:22.900414 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:22.900432 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:22.901447 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:22.901450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:22.902425 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:22.902437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:22.903042 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:22.903045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:22.903205 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:22.903934 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:22.903953 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:22.905488 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:22.907418 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:22.907424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:22.907440 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:22.907782 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:22.908105 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:22.908108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:22.908109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:22.908255 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:22.908803 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:22.908808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:22.909072 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 23:04:22.909233 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:04:22.909542 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:04:22.911717 8447 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33657
I20260810 23:04:22.911986 8447 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:22.912503 8447 heartbeater.cc:511] Master 127.15.56.62:33657 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
I20260810 23:04:22.914194 8159 ts_manager.cc:194] Registered new tserver with Master: 5c4239483857493fad13d728aa0e9cdd (127.15.56.1:35907)
I20260810 23:04:22.915733 8159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:35821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:04:22.930335 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33657
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:23.052385 8452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:23.052719 8452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:23.052781 8452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:23.057009 8452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:23.057085 8452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:23.057148 8452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:04:23.062919 8452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:23.064189 8452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:23.065308 8452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:23.076416 8460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:23.076424 8464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:23.076416 8458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:23.077355 8452 server_base.cc:1034] running on GCE node
I20260810 23:04:23.077950 8452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:23.078837 8452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:23.079980 8452 hybrid_clock.cc:648] HybridClock initialized: now 1786403063079937 us; error 59 us; skew 500 ppm
Aug 10 23:04:23 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403063, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:23.083225 8452 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:04:23.084210 8452 webserver.cc:460] Webserver started at http://127.15.56.2:39775/ using document root <none> and password file <none>
I20260810 23:04:23.084815 8452 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:23.084893 8452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:23.085109 8452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:23.087427 8452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "f5a15d42d77d4c149d8e8ea4192b6d73"
format_stamp: "Formatted at 2026-08-10 23:04:23 on dist-test-slave-6bbx"
server_key: "3e2a34f440e8bcc57a43e9d7b45ed76c"
server_key_iv: "bd145948e275380ac38cdfb5b8eedce6"
server_key_version: "encryptionkey@0"
I20260810 23:04:23.088052 8452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "f5a15d42d77d4c149d8e8ea4192b6d73"
format_stamp: "Formatted at 2026-08-10 23:04:23 on dist-test-slave-6bbx"
server_key: "3e2a34f440e8bcc57a43e9d7b45ed76c"
server_key_iv: "bd145948e275380ac38cdfb5b8eedce6"
server_key_version: "encryptionkey@0"
I20260810 23:04:23.092142 8452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:04:23.094672 8476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:23.096074 8452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:04:23.096177 8452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "f5a15d42d77d4c149d8e8ea4192b6d73"
format_stamp: "Formatted at 2026-08-10 23:04:23 on dist-test-slave-6bbx"
server_key: "3e2a34f440e8bcc57a43e9d7b45ed76c"
server_key_iv: "bd145948e275380ac38cdfb5b8eedce6"
server_key_version: "encryptionkey@0"
I20260810 23:04:23.096287 8452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:23.113827 8452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:23.117761 8452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:23.117970 8452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:23.118661 8452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:23.119576 8452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:23.119628 8452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:23.119709 8452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:23.119743 8452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:23.129686 8452 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:44465
I20260810 23:04:23.129715 8635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:44465 every 8 connection(s)
I20260810 23:04:23.130697 8452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:04:23.138772 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 8452
I20260810 23:04:23.138909 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:04:23.139235 15584 external_mini_cluster.cc:1428] Setting key 14001ede6ac296ef5069c3fd9e74fd46
Aug 10 23:04:23 dist-test-slave-6bbx krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403063, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:23.147006 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.132983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:50265 (local address 127.15.56.62:33657)
0810 23:04:23.133175 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:23.133180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:23.134182 (+ 1002us) server_negotiation.cc:407] Connection header received
0810 23:04:23.135602 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:23.135606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:23.135659 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:23.135766 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:23.137128 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.137689 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.139200 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.139502 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.139601 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.142836 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:23.142845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:23.142847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:23.142876 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:23.144558 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:23.145196 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:23.145201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:23.145203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:23.145307 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:23.145673 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:23.145676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:23.145678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:23.145930 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:23.146029 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:23.146652 (+ 623us) server_negotiation.cc:299] Negotiation successful
0810 23:04:23.146823 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:04:23.148025 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.133453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:50265)
0810 23:04:23.134002 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:23.134053 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:23.135360 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:23.135970 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:23.135979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:23.136386 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:23.136944 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.136969 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.137860 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.137863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.139021 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.139036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.139726 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.139732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.140057 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.140708 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:23.140727 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:23.142590 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:23.144720 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:23.144726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:23.144744 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:23.145075 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:23.145412 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:23.145414 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:23.145416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:23.145584 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:23.146070 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:23.146076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:23.146379 (+ 303us) client_negotiation.cc:769] Sending connection context
0810 23:04:23.146567 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 23:04:23.146949 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":132,"threads_started":1}
I20260810 23:04:23.149523 8636 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33657
I20260810 23:04:23.149802 8636 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:23.150365 8636 heartbeater.cc:511] Master 127.15.56.62:33657 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:04:23.151623 8159 ts_manager.cc:194] Registered new tserver with Master: f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465)
I20260810 23:04:23.152386 8159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:50265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:04:23.170930 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33657
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:23.293427 8644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:23.293768 8644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:23.293835 8644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:23.298054 8644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:23.298141 8644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:23.298228 8644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:04:23.304307 8644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:23.305586 8644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:23.306789 8644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:23.317723 8652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:23.317819 8644 server_base.cc:1034] running on GCE node
W20260810 23:04:23.317723 8654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:23.317723 8651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:23.318456 8644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:23.319212 8644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:23.320389 8644 hybrid_clock.cc:648] HybridClock initialized: now 1786403063320367 us; error 32 us; skew 500 ppm
Aug 10 23:04:23 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403063, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:23.323192 8644 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:04:23.324226 8644 webserver.cc:460] Webserver started at http://127.15.56.3:40237/ using document root <none> and password file <none>
I20260810 23:04:23.324788 8644 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:23.324848 8644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:23.325027 8644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:23.327116 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "60cc09773b3e4a2496083b02d0f15f37"
format_stamp: "Formatted at 2026-08-10 23:04:23 on dist-test-slave-6bbx"
server_key: "4b1f458945b7f77782a681eab060b4cd"
server_key_iv: "ff5bae343a3d2c81a74430f31d438f97"
server_key_version: "encryptionkey@0"
I20260810 23:04:23.327735 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "60cc09773b3e4a2496083b02d0f15f37"
format_stamp: "Formatted at 2026-08-10 23:04:23 on dist-test-slave-6bbx"
server_key: "4b1f458945b7f77782a681eab060b4cd"
server_key_iv: "ff5bae343a3d2c81a74430f31d438f97"
server_key_version: "encryptionkey@0"
I20260810 23:04:23.331710 8644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:04:23.334239 8666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:23.335546 8644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:04:23.335654 8644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "60cc09773b3e4a2496083b02d0f15f37"
format_stamp: "Formatted at 2026-08-10 23:04:23 on dist-test-slave-6bbx"
server_key: "4b1f458945b7f77782a681eab060b4cd"
server_key_iv: "ff5bae343a3d2c81a74430f31d438f97"
server_key_version: "encryptionkey@0"
I20260810 23:04:23.335834 8644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:23.353806 8644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:23.357144 8644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:23.357328 8644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:23.357911 8644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:23.358771 8644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:23.358824 8644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:23.358867 8644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:23.358888 8644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:23.369285 8644 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:40631
I20260810 23:04:23.369314 8831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:40631 every 8 connection(s)
I20260810 23:04:23.370376 8644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:04:23.379443 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 8644
I20260810 23:04:23.379580 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:04:23.379920 15584 external_mini_cluster.cc:1428] Setting key 61356fa36f9ddd5da88cabc09a4a9ee7
Aug 10 23:04:23 dist-test-slave-6bbx krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403063, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:23.385172 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.372545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:59395 (local address 127.15.56.62:33657)
0810 23:04:23.372696 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:23.372700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:23.373643 (+ 943us) server_negotiation.cc:407] Connection header received
0810 23:04:23.375064 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:23.375068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:23.375128 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:23.375205 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:23.376617 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.377192 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.378531 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.378757 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.378860 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.381586 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:23.381589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:23.381591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:23.381618 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:23.383040 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:23.383573 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:23.383576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:23.383577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:23.383659 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:23.384029 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:23.384033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:23.384034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:23.384231 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:23.384320 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:23.384877 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 23:04:23.385007 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:04:23.386070 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.372964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:59395)
0810 23:04:23.373476 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:23.373526 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:23.374832 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:23.375420 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:23.375428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:23.375909 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:23.376453 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.376470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.377332 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.377335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.378392 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.378404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.378918 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.378923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.379105 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.379822 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:23.379841 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:23.381412 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:23.383159 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:23.383166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:23.383186 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:23.383473 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:23.383775 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:23.383778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:23.383780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:23.383937 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:23.384337 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:23.384352 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:23.384624 (+ 272us) client_negotiation.cc:769] Sending connection context
0810 23:04:23.384806 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 23:04:23.385170 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":137,"threads_started":1}
I20260810 23:04:23.387271 8832 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33657
I20260810 23:04:23.387538 8832 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:23.388121 8832 heartbeater.cc:511] Master 127.15.56.62:33657 requested a full tablet report, sending...
I20260810 23:04:23.389243 8159 ts_manager.cc:194] Registered new tserver with Master: 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631)
I20260810 23:04:23.389822 8159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:59395
I20260810 23:04:23.393798 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:04:23.403463 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.395917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.15.56.62:33657)
0810 23:04:23.396035 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:23.396039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:23.396259 (+ 220us) server_negotiation.cc:407] Connection header received
0810 23:04:23.396373 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:23.396377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:23.396423 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:23.396488 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:23.397175 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.397695 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.398536 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.398769 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.398856 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.400263 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:23.400267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:23.400270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:23.400297 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:23.401707 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:23.402159 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:23.402161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:23.402162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:23.402250 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:23.402586 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:23.402589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:23.402591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:23.402846 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:23.402981 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:23.403173 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 23:04:23.403301 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 23:04:23.406630 8159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:04:23.409827 8159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:04:23.424919 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.419090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:35907 (local address 127.0.0.1:45264)
0810 23:04:23.420259 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:23.420270 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:23.420360 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:23.420827 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:23.420834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:23.420859 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:23.421002 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.421010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.422131 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.422134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.423360 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.423369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.424363 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.424367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.424475 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.424512 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:04:23.424623 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 23:04:23.424764 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"thread_start_us":99,"threads_started":1}
I20260810 23:04:23.424937 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.419757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.15.56.1:35907)
0810 23:04:23.420192 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:23.420200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:23.420376 (+ 176us) server_negotiation.cc:407] Connection header received
0810 23:04:23.420468 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:23.420475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:23.420668 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:23.420788 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:23.421115 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.422009 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.423488 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.424248 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.424347 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.424511 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:23.424669 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 23:04:23.424771 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":105,"threads_started":1}
I20260810 23:04:23.425673 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.419767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.15.56.2:44465)
0810 23:04:23.420604 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:23.420635 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:23.420654 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:04:23.420719 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:23.420723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:23.420919 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:23.421079 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:23.421309 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.422443 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.423931 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.424824 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.424930 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.425072 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:23.425414 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 23:04:23.425518 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":142,"threads_started":1}
I20260810 23:04:23.425973 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.418869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:40631 (local address 127.0.0.1:39868)
0810 23:04:23.419763 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:23.419817 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:23.419956 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:23.421855 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:23.421858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:23.421878 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:23.421977 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.421983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.423360 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.423363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.424314 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.424325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.425585 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.425589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.425672 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.425694 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:04:23.425763 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 23:04:23.425832 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":102,"threads_started":1}
I20260810 23:04:23.426472 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.419021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.15.56.3:40631)
0810 23:04:23.421401 (+ 2380us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:23.421413 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:23.421436 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:04:23.421511 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:23.421516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:23.421702 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:23.421853 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:23.422096 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.423218 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.424461 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.425261 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.425353 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.425519 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:23.426160 (+ 641us) server_negotiation.cc:299] Negotiation successful
0810 23:04:23.426285 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2266,"thread_start_us":107,"threads_started":1}
I20260810 23:04:23.426508 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.419186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:44465 (local address 127.0.0.1:40086)
0810 23:04:23.420310 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:23.420322 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:23.420437 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:23.421065 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:23.421069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:23.421089 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:23.421184 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.421191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.422595 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.422598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.423800 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.423811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.425141 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.425145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.425233 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.425256 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:04:23.425861 (+ 605us) client_negotiation.cc:240] Negotiation successful
0810 23:04:23.425934 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260810 23:04:23.428440 8344 tablet_service.cc:1467] Processing CreateTablet for tablet aceb2d93ef574a778d178ce6c552f84c (DEFAULT_TABLE table=test-table [id=7f76242bf079431ebf3a34b05df6fbb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:23.429688 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aceb2d93ef574a778d178ce6c552f84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:23.429548 8540 tablet_service.cc:1467] Processing CreateTablet for tablet aceb2d93ef574a778d178ce6c552f84c (DEFAULT_TABLE table=test-table [id=7f76242bf079431ebf3a34b05df6fbb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:23.429484 8731 tablet_service.cc:1467] Processing CreateTablet for tablet aceb2d93ef574a778d178ce6c552f84c (DEFAULT_TABLE table=test-table [id=7f76242bf079431ebf3a34b05df6fbb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:23.430656 8540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aceb2d93ef574a778d178ce6c552f84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:23.430660 8731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aceb2d93ef574a778d178ce6c552f84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:23.436908 8858 tablet_bootstrap.cc:492] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: Bootstrap starting.
I20260810 23:04:23.438122 8860 tablet_bootstrap.cc:492] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Bootstrap starting.
I20260810 23:04:23.439826 8858 tablet_bootstrap.cc:654] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:23.440021 8861 tablet_bootstrap.cc:492] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: Bootstrap starting.
I20260810 23:04:23.441699 8860 tablet_bootstrap.cc:654] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:23.442219 8858 log.cc:824] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:23.443151 8860 log.cc:824] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:23.444614 8858 tablet_bootstrap.cc:492] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: No bootstrap required, opened a new log
I20260810 23:04:23.444655 8861 tablet_bootstrap.cc:654] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:23.444828 8858 ts_tablet_manager.cc:1397] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 23:04:23.445482 8860 tablet_bootstrap.cc:492] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: No bootstrap required, opened a new log
I20260810 23:04:23.445696 8860 ts_tablet_manager.cc:1397] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 23:04:23.446125 8861 log.cc:824] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:23.448385 8861 tablet_bootstrap.cc:492] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: No bootstrap required, opened a new log
I20260810 23:04:23.448588 8861 ts_tablet_manager.cc:1397] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 23:04:23.448572 8858 raft_consensus.cc:348] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } }
I20260810 23:04:23.448892 8858 raft_consensus.cc:374] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:23.448951 8858 raft_consensus.cc:729] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a15d42d77d4c149d8e8ea4192b6d73, State: Initialized, Role: FOLLOWER
I20260810 23:04:23.449445 8858 consensus_queue.cc:260] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } }
I20260810 23:04:23.450436 8858 ts_tablet_manager.cc:1428] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:04:23.450472 8860 raft_consensus.cc:348] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } }
I20260810 23:04:23.450754 8860 raft_consensus.cc:374] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:23.450806 8860 raft_consensus.cc:729] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4239483857493fad13d728aa0e9cdd, State: Initialized, Role: FOLLOWER
I20260810 23:04:23.450826 8636 heartbeater.cc:503] Master 127.15.56.62:33657 was elected leader, sending a full tablet report...
I20260810 23:04:23.451311 8860 consensus_queue.cc:260] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } }
I20260810 23:04:23.452037 8447 heartbeater.cc:503] Master 127.15.56.62:33657 was elected leader, sending a full tablet report...
I20260810 23:04:23.452630 8860 ts_tablet_manager.cc:1428] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Time spent starting tablet: real 0.007s user 0.001s sys 0.007s
I20260810 23:04:23.452641 8861 raft_consensus.cc:348] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } }
I20260810 23:04:23.452916 8861 raft_consensus.cc:374] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:23.452976 8861 raft_consensus.cc:729] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60cc09773b3e4a2496083b02d0f15f37, State: Initialized, Role: FOLLOWER
I20260810 23:04:23.453408 8861 consensus_queue.cc:260] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } }
I20260810 23:04:23.454025 8832 heartbeater.cc:503] Master 127.15.56.62:33657 was elected leader, sending a full tablet report...
I20260810 23:04:23.454233 8861 ts_tablet_manager.cc:1428] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
W20260810 23:04:23.621528 8836 tablet.cc:2369] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:04:23.625300 8867 raft_consensus.cc:482] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:04:23.625517 8867 raft_consensus.cc:504] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } }
I20260810 23:04:23.626705 8867 leader_election.cc:288] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631), 5c4239483857493fad13d728aa0e9cdd (127.15.56.1:35907)
I20260810 23:04:23.630437 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.627004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:35907 (local address 127.15.56.2:33869)
0810 23:04:23.627139 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:23.627154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:23.627266 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:23.627566 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:23.627569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:23.627597 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:23.627727 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.627733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.628462 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.628466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.629250 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.629260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.630003 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.630007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.630110 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.630129 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 23:04:23.630196 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 23:04:23.630275 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":2,"spinlock_wait_cycles":896}
I20260810 23:04:23.630436 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.627212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:33869 (local address 127.15.56.1:35907)
0810 23:04:23.627357 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:23.627361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:23.627374 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:04:23.627424 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:23.627426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:23.627466 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:23.627530 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:23.627845 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.628352 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.629383 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.629877 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.629976 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.630043 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:23.630223 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 23:04:23.630298 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:04:23.630472 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.627122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:46129 (local address 127.15.56.3:40631)
0810 23:04:23.627255 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:23.627259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:23.627562 (+ 303us) server_negotiation.cc:407] Connection header received
0810 23:04:23.627707 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:23.627711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:23.627756 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:23.627820 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:23.628064 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.628555 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.629561 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.629955 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.630043 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.630104 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:23.630284 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 23:04:23.630355 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:04:23.630645 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.627009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:40631 (local address 127.15.56.2:46129)
0810 23:04:23.627462 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:23.627473 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:23.627586 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:23.627839 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:23.627841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:23.627857 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:23.627964 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.627969 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.628654 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.628657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.629440 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:23.629449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.630062 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.630065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:23.630169 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.630187 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 23:04:23.630252 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:04:23.630323 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":141,"thread_start_us":143,"threads_started":1}
I20260810 23:04:23.631213 8376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aceb2d93ef574a778d178ce6c552f84c" candidate_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c4239483857493fad13d728aa0e9cdd" is_pre_election: true
I20260810 23:04:23.631294 8762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aceb2d93ef574a778d178ce6c552f84c" candidate_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60cc09773b3e4a2496083b02d0f15f37" is_pre_election: true
I20260810 23:04:23.631592 8762 raft_consensus.cc:2454] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5a15d42d77d4c149d8e8ea4192b6d73 in term 0.
I20260810 23:04:23.631592 8376 raft_consensus.cc:2454] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5a15d42d77d4c149d8e8ea4192b6d73 in term 0.
W20260810 23:04:23.632069 8637 tablet.cc:2369] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:04:23.632155 8477 leader_election.cc:302] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60cc09773b3e4a2496083b02d0f15f37, f5a15d42d77d4c149d8e8ea4192b6d73; no voters:
I20260810 23:04:23.632445 8867 raft_consensus.cc:2781] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:04:23.632539 8867 raft_consensus.cc:482] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:04:23.632591 8867 raft_consensus.cc:3037] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:23.633688 8867 raft_consensus.cc:504] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } }
I20260810 23:04:23.634038 8867 leader_election.cc:288] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [CANDIDATE]: Term 1 election: Requested vote from peers 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631), 5c4239483857493fad13d728aa0e9cdd (127.15.56.1:35907)
I20260810 23:04:23.634411 8762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aceb2d93ef574a778d178ce6c552f84c" candidate_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60cc09773b3e4a2496083b02d0f15f37"
I20260810 23:04:23.634408 8376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aceb2d93ef574a778d178ce6c552f84c" candidate_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c4239483857493fad13d728aa0e9cdd"
I20260810 23:04:23.634532 8376 raft_consensus.cc:3037] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:23.634528 8762 raft_consensus.cc:3037] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:23.635624 8376 raft_consensus.cc:2454] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5a15d42d77d4c149d8e8ea4192b6d73 in term 1.
I20260810 23:04:23.635725 8762 raft_consensus.cc:2454] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5a15d42d77d4c149d8e8ea4192b6d73 in term 1.
I20260810 23:04:23.636015 8481 leader_election.cc:302] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c4239483857493fad13d728aa0e9cdd, f5a15d42d77d4c149d8e8ea4192b6d73; no voters:
I20260810 23:04:23.636204 8867 raft_consensus.cc:2781] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:23.636436 8867 raft_consensus.cc:686] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 1 LEADER]: Becoming Leader. State: Replica: f5a15d42d77d4c149d8e8ea4192b6d73, State: Running, Role: LEADER
I20260810 23:04:23.636807 8867 consensus_queue.cc:237] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } }
I20260810 23:04:23.640182 8159 catalog_manager.cc:5495] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2). New cstate: current_term: 1 leader_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } health_report { overall_health: HEALTHY } } }
W20260810 23:04:23.645504 8448 tablet.cc:2369] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:04:23.662781 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.659198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.15.56.2:44465)
0810 23:04:23.659343 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:23.659346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:23.659360 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:04:23.659444 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:23.659446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:23.659501 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:23.659590 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:04:23.659894 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.660428 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.661379 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:23.661603 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:23.661686 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:23.661976 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:04:23.662471 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:04:23.662575 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:23.662606 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 23:04:23.662661 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:04:23.669132 8376 raft_consensus.cc:1261] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Refusing update from remote peer f5a15d42d77d4c149d8e8ea4192b6d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:23.669133 8762 raft_consensus.cc:1261] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37 [term 1 FOLLOWER]: Refusing update from remote peer f5a15d42d77d4c149d8e8ea4192b6d73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:23.669878 8867 consensus_queue.cc:1035] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:23.670071 8875 consensus_queue.cc:1035] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:23.679121 8881 mvcc.cc:205] Tried to move back new op lower bound from 7317106948780617728 to 7317106948659322880. Current Snapshot: MvccSnapshot[applied={T|T < 7317106948780617728}]
I20260810 23:04:23.687863 8157 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:23.688033 8157 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:23.690595 8157 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Sending DeleteTablet for 3 replicas of tablet aceb2d93ef574a778d178ce6c552f84c
I20260810 23:04:23.691434 8731 tablet_service.cc:1514] Processing DeleteTablet for tablet aceb2d93ef574a778d178ce6c552f84c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:23 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:39868
I20260810 23:04:23.691506 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet aceb2d93ef574a778d178ce6c552f84c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:23 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45264
I20260810 23:04:23.692003 8540 tablet_service.cc:1514] Processing DeleteTablet for tablet aceb2d93ef574a778d178ce6c552f84c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:23 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:40086
I20260810 23:04:23.692037 8894 tablet_replica.cc:330] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: stopping tablet replica
I20260810 23:04:23.692392 8894 raft_consensus.cc:2229] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:23.692458 8896 tablet_replica.cc:330] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: stopping tablet replica
I20260810 23:04:23.692658 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 8251
I20260810 23:04:23.692773 8894 raft_consensus.cc:2258] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:23.692986 8896 raft_consensus.cc:2229] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:04:23.693213 8892 tablet_replica.cc:330] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: stopping tablet replica
I20260810 23:04:23.693493 8892 raft_consensus.cc:2229] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:23.693660 8896 raft_consensus.cc:2258] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:23.693760 8892 raft_consensus.cc:2258] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:23.696972 8892 ts_tablet_manager.cc:1905] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:23.698180 8896 ts_tablet_manager.cc:1905] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:23.700595 8892 ts_tablet_manager.cc:1918] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:23.700826 8892 log.cc:1196] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/aceb2d93ef574a778d178ce6c552f84c
I20260810 23:04:23.701126 8896 ts_tablet_manager.cc:1918] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:23.701239 8896 log.cc:1196] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/aceb2d93ef574a778d178ce6c552f84c
I20260810 23:04:23.701489 8892 ts_tablet_manager.cc:1939] T aceb2d93ef574a778d178ce6c552f84c P 60cc09773b3e4a2496083b02d0f15f37: Deleting consensus metadata
I20260810 23:04:23.701592 8896 ts_tablet_manager.cc:1939] T aceb2d93ef574a778d178ce6c552f84c P f5a15d42d77d4c149d8e8ea4192b6d73: Deleting consensus metadata
I20260810 23:04:23.703234 8141 catalog_manager.cc:4841] TS 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631): tablet aceb2d93ef574a778d178ce6c552f84c (table test-table [id=7f76242bf079431ebf3a34b05df6fbb8]) successfully deleted
I20260810 23:04:23.708340 8143 catalog_manager.cc:4841] TS f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465): tablet aceb2d93ef574a778d178ce6c552f84c (table test-table [id=7f76242bf079431ebf3a34b05df6fbb8]) successfully deleted
W20260810 23:04:23.714432 8144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:35907 (error 108)
I20260810 23:04:23.714672 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 8452
I20260810 23:04:23.722692 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:23.722357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:35907 (local address 127.0.0.1:45280)
0810 23:04:23.722513 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:23.722597 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:35907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 23:04:23.722905 8144 catalog_manager.cc:4584] TS 5c4239483857493fad13d728aa0e9cdd (127.15.56.1:35907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aceb2d93ef574a778d178ce6c552f84c: Network error: Client connection negotiation failed: client connection to 127.15.56.1:35907: connect: Connection refused (error 111)
I20260810 23:04:23.728451 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 8644
I20260810 23:04:23.746570 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 8120
I20260810 23:04:43.761415 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33657
--webserver_interface=127.15.56.62
--webserver_port=45279
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:33657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:43.881412 8900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:43.881723 8900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:43.881769 8900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:43.886011 8900 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:04:43.886085 8900 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:04:43.886102 8900 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:04:43.886114 8900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:04:43.886132 8900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:43.886149 8900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:43.886170 8900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:04:43.886184 8900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:04:43.892402 8900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:33657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:43.893646 8900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:43.894886 8900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:43.909998 8909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:43.910022 8914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:43.910104 8900 server_base.cc:1034] running on GCE node
W20260810 23:04:43.910022 8916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:43.910971 8900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:43.912365 8900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:43.913533 8900 hybrid_clock.cc:648] HybridClock initialized: now 1786403083913501 us; error 51 us; skew 500 ppm
Aug 10 23:04:43 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403083, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:43.916710 8900 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:04:43.917778 8900 webserver.cc:460] Webserver started at http://127.15.56.62:45279/ using document root <none> and password file <none>
I20260810 23:04:43.918398 8900 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:43.918491 8900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:43.922755 8900 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:04:43.925273 8926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:43.926855 8900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:04:43.927013 8900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "123e3d443f6946b4a052db96b8313ee9"
format_stamp: "Formatted at 2026-08-10 23:04:22 on dist-test-slave-6bbx"
server_key: "a6296df0d28b5953ffb8900d4db4bc42"
server_key_iv: "70b977ff14b6611bdce4f4a8dbdf9507"
server_key_version: "encryptionkey@0"
I20260810 23:04:43.927748 8900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:43.944192 8900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:43.947557 8900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:43.947793 8900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:43.955263 8900 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:33657
I20260810 23:04:43.955535 9008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:33657 every 8 connection(s)
I20260810 23:04:43.956516 8900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:04:43.959864 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 8900
I20260810 23:04:43.960392 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:35907
--local_ip_for_outbound_sockets=127.15.56.1
--tserver_master_addrs=127.15.56.62:33657
--webserver_port=45271
--webserver_interface=127.15.56.1
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:04:43.961747 9009 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:04:43.963778 9009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Bootstrap starting.
I20260810 23:04:43.983964 9009 log.cc:824] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:43.993395 9009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:04:43.993841 9009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Bootstrap complete.
I20260810 23:04:43.998054 9009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:04:43.998486 9009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123e3d443f6946b4a052db96b8313ee9, State: Initialized, Role: FOLLOWER
I20260810 23:04:43.999011 9009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:04:43.999133 9009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:04:43.999199 9009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:04:43.999300 9009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:04:44.001444 9009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:04:44.001823 9009 leader_election.cc:302] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 123e3d443f6946b4a052db96b8313ee9; no voters:
I20260810 23:04:44.002175 9009 leader_election.cc:288] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:04:44.002369 9018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:04:44.002646 9018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 2 LEADER]: Becoming Leader. State: Replica: 123e3d443f6946b4a052db96b8313ee9, State: Running, Role: LEADER
I20260810 23:04:44.003006 9018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:04:44.003496 9009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:04:44.004712 9019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "123e3d443f6946b4a052db96b8313ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } } }
I20260810 23:04:44.004860 9019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:44.005125 9020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 123e3d443f6946b4a052db96b8313ee9. Latest consensus state: current_term: 2 leader_uuid: "123e3d443f6946b4a052db96b8313ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } } }
I20260810 23:04:44.005242 9020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:44.005357 9030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:04:44.010862 9030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:04:44.011637 9030 catalog_manager.cc:1259] Loaded cluster ID: 296558d7dbb7434d854158f5673db1ba
I20260810 23:04:44.011729 9030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:04:44.014295 9030 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:04:44.015024 9030 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Loaded TSK: 0
I20260810 23:04:44.033970 9030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Generated new TSK 1
I20260810 23:04:44.034329 9030 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:04:44.096567 9014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:44.097105 9014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:44.097178 9014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:44.103066 9014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:44.103147 9014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:44.103212 9014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:04:44.109105 9014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:35907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=45271
--tserver_master_addrs=127.15.56.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:44.110291 9014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:44.111402 9014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:44.122673 9050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:44.122673 9049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:44.122840 9014 server_base.cc:1034] running on GCE node
W20260810 23:04:44.122673 9052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:44.123495 9014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:44.124338 9014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:44.125576 9014 hybrid_clock.cc:648] HybridClock initialized: now 1786403084125553 us; error 34 us; skew 500 ppm
Aug 10 23:04:44 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403084, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:44.128885 9014 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:04:44.130048 9014 webserver.cc:460] Webserver started at http://127.15.56.1:45271/ using document root <none> and password file <none>
I20260810 23:04:44.130667 9014 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:44.130771 9014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:44.135257 9014 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:04:44.137717 9068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:44.139168 9014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:04:44.139305 9014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "5c4239483857493fad13d728aa0e9cdd"
format_stamp: "Formatted at 2026-08-10 23:04:22 on dist-test-slave-6bbx"
server_key: "4b6a7c027b1283fcbd312401b15739a8"
server_key_iv: "fa68cfac543e83c6102bf9b41a3bc44e"
server_key_version: "encryptionkey@0"
I20260810 23:04:44.139989 9014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:44.161353 9014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:44.164994 9014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:44.165215 9014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:44.165899 9014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:44.167100 9079 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:04:44.171710 9014 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:04:44.171794 9014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 23:04:44.171869 9014 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:04:44.173693 9014 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:04:44.173743 9014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 23:04:44.174108 9079 tablet_bootstrap.cc:492] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Bootstrap starting.
I20260810 23:04:44.183925 9014 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:35907
I20260810 23:04:44.183997 9250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:35907 every 8 connection(s)
I20260810 23:04:44.184973 9014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:04:44.190888 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 9014
I20260810 23:04:44.191427 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:44465
--local_ip_for_outbound_sockets=127.15.56.2
--tserver_master_addrs=127.15.56.62:33657
--webserver_port=39775
--webserver_interface=127.15.56.2
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:04:44.193373 9079 log.cc:824] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:04:44 dist-test-slave-6bbx krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403084, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:44.201866 9079 tablet_bootstrap.cc:492] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:04:44.202366 9079 tablet_bootstrap.cc:492] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Bootstrap complete.
I20260810 23:04:44.203699 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.187379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:35317)
0810 23:04:44.187971 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:44.188017 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:44.188845 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:44.190032 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:44.190041 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:44.190480 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:44.190983 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.191002 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.192476 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.192479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.193541 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.193553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.194468 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.194472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.194675 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.195286 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:44.195305 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:44.197228 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:44.200012 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:44.200018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:44.200031 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:44.200374 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:44.200793 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:44.200797 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:44.200798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:44.200964 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:44.201727 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:44.201733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:44.202036 (+ 303us) client_negotiation.cc:769] Sending connection context
0810 23:04:44.202278 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 23:04:44.202640 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":192,"threads_started":1}
I20260810 23:04:44.203922 9079 ts_tablet_manager.cc:1397] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.008s
I20260810 23:04:44.204454 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.187364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:35317 (local address 127.15.56.62:33657)
0810 23:04:44.187773 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.187796 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.188150 (+ 354us) server_negotiation.cc:407] Connection header received
0810 23:04:44.189292 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.189304 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.189622 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.189939 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:44.191175 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.192338 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.193678 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.194273 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.194486 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.197507 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:44.197514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:44.197533 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:44.197571 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:44.199865 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.200502 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.200509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.200515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.200689 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.201067 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.201071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.201072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.201594 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:44.201747 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:44.202577 (+ 830us) server_negotiation.cc:299] Negotiation successful
0810 23:04:44.202981 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
I20260810 23:04:44.206684 9079 raft_consensus.cc:348] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } }
I20260810 23:04:44.207187 9079 raft_consensus.cc:729] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4239483857493fad13d728aa0e9cdd, State: Initialized, Role: FOLLOWER
I20260810 23:04:44.207249 9254 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33657
I20260810 23:04:44.207495 9254 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:44.207746 9079 consensus_queue.cc:260] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } } peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } }
I20260810 23:04:44.208117 9254 heartbeater.cc:511] Master 127.15.56.62:33657 requested a full tablet report, sending...
I20260810 23:04:44.208703 9079 ts_tablet_manager.cc:1428] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:04:44.210978 8953 ts_manager.cc:194] Registered new tserver with Master: 5c4239483857493fad13d728aa0e9cdd (127.15.56.1:35907)
I20260810 23:04:44.215602 8953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:35317
I20260810 23:04:44.218466 9254 heartbeater.cc:503] Master 127.15.56.62:33657 was elected leader, sending a full tablet report...
Aug 10 23:04:44 dist-test-slave-6bbx krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403083, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.1@KRBTEST.COM
I20260810 23:04:44.224017 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.214449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:35907 (local address 127.0.0.1:45442)
0810 23:04:44.214847 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:44.214895 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:44.215020 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:44.215635 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:44.215641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:44.216089 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:44.216195 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.216202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.217225 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.217229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.217900 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.217911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.218458 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.218461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.218617 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.219215 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:44.219227 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:44.220588 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:44.222313 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:44.222323 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:44.222330 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:44.222553 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:44.222837 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:44.222840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:44.222842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:44.222921 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:44.223352 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:44.223358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:44.223477 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 23:04:44.223597 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 23:04:44.223835 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260810 23:04:44.224037 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.214542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.15.56.1:35907)
0810 23:04:44.215243 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.215252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.215270 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:04:44.215330 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.215334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.215486 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.215610 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:44.216316 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.217115 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.218012 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.218345 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.218440 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.220722 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:44.220731 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:44.220742 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:44.220765 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:44.222208 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.222658 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.222663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.222664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.222744 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.223002 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.223006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.223008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.223239 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:44.223368 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:44.223732 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 23:04:44.223899 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":147,"threads_started":1}
I20260810 23:04:44.224984 9137 tablet_service.cc:1514] Processing DeleteTablet for tablet aceb2d93ef574a778d178ce6c552f84c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:23 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45442
I20260810 23:04:44.225865 9267 tablet_replica.cc:330] stopping tablet replica
I20260810 23:04:44.226207 9267 raft_consensus.cc:2229] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:44.226486 9267 raft_consensus.cc:2258] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:44.227725 9267 ts_tablet_manager.cc:1905] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:44.232651 9267 ts_tablet_manager.cc:1918] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:44.232788 9267 log.cc:1196] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal/wals/aceb2d93ef574a778d178ce6c552f84c
I20260810 23:04:44.233096 9267 ts_tablet_manager.cc:1939] T aceb2d93ef574a778d178ce6c552f84c P 5c4239483857493fad13d728aa0e9cdd: Deleting consensus metadata
I20260810 23:04:44.235216 8930 catalog_manager.cc:4841] TS 5c4239483857493fad13d728aa0e9cdd (127.15.56.1:35907): tablet aceb2d93ef574a778d178ce6c552f84c (table test-table [id=7f76242bf079431ebf3a34b05df6fbb8]) successfully deleted
W20260810 23:04:44.320393 9262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:44.320665 9262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:44.320710 9262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:44.324781 9262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:44.324843 9262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:44.324913 9262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:04:44.330412 9262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:44465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=39775
--tserver_master_addrs=127.15.56.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:44.331625 9262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:44.332794 9262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:44.342804 9275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:44.342922 9262 server_base.cc:1034] running on GCE node
W20260810 23:04:44.342818 9273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:44.342804 9277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:44.343634 9262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:44.344334 9262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:44.345512 9262 hybrid_clock.cc:648] HybridClock initialized: now 1786403084345480 us; error 43 us; skew 500 ppm
Aug 10 23:04:44 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403084, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:44.348408 9262 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:04:44.349323 9262 webserver.cc:460] Webserver started at http://127.15.56.2:39775/ using document root <none> and password file <none>
I20260810 23:04:44.349879 9262 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:44.349957 9262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:44.354074 9262 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:04:44.356371 9286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:44.357704 9262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:04:44.357831 9262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "f5a15d42d77d4c149d8e8ea4192b6d73"
format_stamp: "Formatted at 2026-08-10 23:04:23 on dist-test-slave-6bbx"
server_key: "3e2a34f440e8bcc57a43e9d7b45ed76c"
server_key_iv: "bd145948e275380ac38cdfb5b8eedce6"
server_key_version: "encryptionkey@0"
I20260810 23:04:44.358448 9262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:44.371886 9262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:44.374872 9262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:44.375056 9262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:44.375701 9262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:44.376552 9262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:44.376602 9262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:44.376657 9262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:44.376688 9262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:44.385459 9262 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:44465
I20260810 23:04:44.385488 9448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:44465 every 8 connection(s)
I20260810 23:04:44.386511 9262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:04:44.390399 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 9262
I20260810 23:04:44.390833 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:40631
--local_ip_for_outbound_sockets=127.15.56.3
--tserver_master_addrs=127.15.56.62:33657
--webserver_port=40237
--webserver_interface=127.15.56.3
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:04:44 dist-test-slave-6bbx krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403084, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:44.402173 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.388511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:45447 (local address 127.15.56.62:33657)
0810 23:04:44.388656 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.388659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.389601 (+ 942us) server_negotiation.cc:407] Connection header received
0810 23:04:44.391046 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.391050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.391102 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.391146 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:44.392774 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.393329 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.394724 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.395103 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.395226 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.398045 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:44.398048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:44.398050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:44.398082 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:44.399552 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.400238 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.400241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.400243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.400332 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.400789 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.400793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.400794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.401004 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:44.401091 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:44.401889 (+ 798us) server_negotiation.cc:299] Negotiation successful
0810 23:04:44.402023 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:04:44.403399 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.388915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:45447)
0810 23:04:44.389435 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:44.389483 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:44.390792 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:44.391514 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:44.391521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:44.391956 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:44.392590 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.392611 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.393461 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.393465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.394566 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.394578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.395289 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.395292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.395446 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.396106 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:44.396124 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:44.397845 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:44.399713 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:44.399720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:44.399740 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:44.400117 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:44.400427 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:44.400430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:44.400432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:44.400650 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:44.401115 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:44.401122 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:44.401501 (+ 379us) client_negotiation.cc:769] Sending connection context
0810 23:04:44.401760 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 23:04:44.402184 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":123,"threads_started":1}
I20260810 23:04:44.404791 9449 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33657
I20260810 23:04:44.405074 9449 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:44.405697 9449 heartbeater.cc:511] Master 127.15.56.62:33657 requested a full tablet report, sending...
I20260810 23:04:44.406816 8953 ts_manager.cc:194] Registered new tserver with Master: f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465)
I20260810 23:04:44.407397 8953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:45447
W20260810 23:04:44.511001 9453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:44.511291 9453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:44.511355 9453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:44.515385 9453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:44.515465 9453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:44.515548 9453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:04:44.521132 9453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:40631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=40237
--tserver_master_addrs=127.15.56.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:04:44.522325 9453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:44.523429 9453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:44.533921 9463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:44.533967 9453 server_base.cc:1034] running on GCE node
W20260810 23:04:44.533907 9465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:44.533907 9468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:44.534631 9453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:44.535521 9453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:44.536717 9453 hybrid_clock.cc:648] HybridClock initialized: now 1786403084536683 us; error 50 us; skew 500 ppm
Aug 10 23:04:44 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403084, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:44.539954 9453 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:04:44.540951 9453 webserver.cc:460] Webserver started at http://127.15.56.3:40237/ using document root <none> and password file <none>
I20260810 23:04:44.541517 9453 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:44.541594 9453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:44.545836 9453 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:04:44.548018 9481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:44.549341 9453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:04:44.549446 9453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "60cc09773b3e4a2496083b02d0f15f37"
format_stamp: "Formatted at 2026-08-10 23:04:23 on dist-test-slave-6bbx"
server_key: "4b1f458945b7f77782a681eab060b4cd"
server_key_iv: "ff5bae343a3d2c81a74430f31d438f97"
server_key_version: "encryptionkey@0"
I20260810 23:04:44.550015 9453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:44.562770 9453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:44.566341 9453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:44.566540 9453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:44.567205 9453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:44.568133 9453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:44.568192 9453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:44.568240 9453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:44.568270 9453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:44.577346 9453 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:40631
I20260810 23:04:44.577425 9648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:40631 every 8 connection(s)
I20260810 23:04:44.578450 9453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:04:44.579259 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 9453
I20260810 23:04:44.588696 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.580035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.15.56.62:33657)
0810 23:04:44.580189 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.580193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.580463 (+ 270us) server_negotiation.cc:407] Connection header received
0810 23:04:44.580614 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.580617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.580678 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.580787 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:44.581680 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.582289 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.583477 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.583826 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.583909 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.585456 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:44.585460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:44.585463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:44.585491 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:44.586923 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.587345 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.587348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.587350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.587448 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.587800 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.587803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.587804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.587997 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:44.588265 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:44.588389 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 23:04:44.588522 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 23:04:44 dist-test-slave-6bbx krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403084, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:04:44.593659 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.580453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:34157 (local address 127.15.56.62:33657)
0810 23:04:44.580717 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.580720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.581746 (+ 1026us) server_negotiation.cc:407] Connection header received
0810 23:04:44.583383 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.583386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.583439 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.583543 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:44.585112 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.585766 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.587070 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.587305 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.587406 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.590428 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:44.590432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:44.590434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:44.590457 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:44.591768 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.592261 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.592264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.592265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.592334 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.592616 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.592618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.592619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.592784 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:44.592854 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:44.593388 (+ 534us) server_negotiation.cc:299] Negotiation successful
0810 23:04:44.593522 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"spinlock_wait_cycles":896,"thread_start_us":94,"threads_started":1}
I20260810 23:04:44.594503 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.581030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:34157)
0810 23:04:44.581584 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:44.581648 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:44.583114 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:44.583749 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:44.583756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:44.584265 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:44.584939 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.584961 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.585900 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.585904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.586944 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.586955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.587453 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.587456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.587719 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.588303 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:44.588325 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:44.590245 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:44.591883 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:44.591889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:44.591903 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:44.592169 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:44.592409 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:44.592411 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:44.592412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:44.592549 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:44.592866 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:44.592871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:44.593127 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 23:04:44.593300 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 23:04:44.593574 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":128,"threads_started":1}
I20260810 23:04:44.595808 9649 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33657
I20260810 23:04:44.596064 9649 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:44.596590 9649 heartbeater.cc:511] Master 127.15.56.62:33657 requested a full tablet report, sending...
I20260810 23:04:44.597811 8953 ts_manager.cc:194] Registered new tserver with Master: 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631)
I20260810 23:04:44.598547 8953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:34157
I20260810 23:04:44.603031 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:04:44.607301 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.603802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.15.56.62:33657)
0810 23:04:44.603955 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.603959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.604225 (+ 266us) server_negotiation.cc:407] Connection header received
0810 23:04:44.604353 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.604355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.604467 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.604563 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:04:44.604856 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.605442 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.606294 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.606515 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.606587 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.606821 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:04:44.606972 (+ 151us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 23:04:44.607185 (+ 213us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40682: token expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 23:04:44.607376 9654 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40682: token expired
W20260810 23:04:44.607811 8840 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.56.62:33657)
I20260810 23:04:44.614569 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.608155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.15.56.62:33657)
0810 23:04:44.608270 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.608274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.608289 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:04:44.608382 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.608384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.608420 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.608480 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:44.609056 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.609552 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.610335 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.610538 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.610640 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.611999 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:44.612002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:44.612004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:44.612028 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:44.613233 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.613618 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.613620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.613621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.613691 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:44.613886 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:44.613888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:44.613889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:44.614040 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:44.614143 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:44.614356 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 23:04:44.614450 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:04:44.617067 8953 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:04:44.620352 8953 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:04:44.627940 9137 tablet_service.cc:1467] Processing CreateTablet for tablet f34fa85e141a41849646bc46bb61893c (DEFAULT_TABLE table=test-table [id=2c2d04c9202742d9a5fb634ac76eb66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:44.628566 9137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f34fa85e141a41849646bc46bb61893c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:44.631349 9079 tablet_bootstrap.cc:492] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: Bootstrap starting.
I20260810 23:04:44.632362 9079 tablet_bootstrap.cc:654] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:44.632730 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.627314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:40631 (local address 127.0.0.1:46380)
0810 23:04:44.627698 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:44.627711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:44.627795 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:44.628553 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:44.628557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:44.628576 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:44.628677 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.628684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.629990 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.629993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.631093 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.631100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.632316 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.632319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.632431 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.632484 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 23:04:44.632546 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 23:04:44.632614 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":4,"thread_start_us":121,"threads_started":1}
I20260810 23:04:44.633427 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.627697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.15.56.3:40631)
0810 23:04:44.628065 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.628077 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.628100 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:04:44.628185 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.628192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.628403 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.628564 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:44.628818 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.629869 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.631201 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.632187 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.632984 (+ 797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.633119 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:44.633172 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 23:04:44.633270 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":158,"threads_started":1}
I20260810 23:04:44.633891 9079 tablet_bootstrap.cc:492] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: No bootstrap required, opened a new log
I20260810 23:04:44.633971 9079 ts_tablet_manager.cc:1397] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:04:44.634378 9079 raft_consensus.cc:348] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:04:44.634509 9079 raft_consensus.cc:374] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:44.634547 9079 raft_consensus.cc:729] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4239483857493fad13d728aa0e9cdd, State: Initialized, Role: FOLLOWER
I20260810 23:04:44.634714 9079 consensus_queue.cc:260] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:04:44.634995 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.626804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:44465 (local address 127.0.0.1:43268)
0810 23:04:44.626920 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:44.626936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:44.627009 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:44.628013 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:44.628016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:44.628034 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:44.628141 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.628147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.629704 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.629707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.630907 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.630918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.634569 (+ 3651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.634572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.634679 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.634702 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:04:44.634782 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:04:44.634875 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 23:04:44.635057 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.627090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.15.56.2:44465)
0810 23:04:44.627424 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.627440 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.627464 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:04:44.627548 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.627553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.627835 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.628298 (+ 463us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:44.628489 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.629589 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.631058 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.634441 (+ 3383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.634552 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.634716 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:44.634816 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 23:04:44.634912 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":95,"threads_started":1}
I20260810 23:04:44.635156 9079 ts_tablet_manager.cc:1428] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:04:44.636722 9551 tablet_service.cc:1467] Processing CreateTablet for tablet f34fa85e141a41849646bc46bb61893c (DEFAULT_TABLE table=test-table [id=2c2d04c9202742d9a5fb634ac76eb66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:44.637753 9551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f34fa85e141a41849646bc46bb61893c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:44.637871 9354 tablet_service.cc:1467] Processing CreateTablet for tablet f34fa85e141a41849646bc46bb61893c (DEFAULT_TABLE table=test-table [id=2c2d04c9202742d9a5fb634ac76eb66c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:44.638727 9354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f34fa85e141a41849646bc46bb61893c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:44.645054 9668 tablet_bootstrap.cc:492] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37: Bootstrap starting.
I20260810 23:04:44.645749 9672 tablet_bootstrap.cc:492] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73: Bootstrap starting.
I20260810 23:04:44.648171 9668 tablet_bootstrap.cc:654] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:44.648408 9672 tablet_bootstrap.cc:654] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:44.649155 9668 log.cc:824] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:44.649155 9672 log.cc:824] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:44.651418 9672 tablet_bootstrap.cc:492] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73: No bootstrap required, opened a new log
I20260810 23:04:44.651418 9668 tablet_bootstrap.cc:492] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37: No bootstrap required, opened a new log
I20260810 23:04:44.651641 9672 ts_tablet_manager.cc:1397] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:04:44.651643 9668 ts_tablet_manager.cc:1397] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 23:04:44.655709 9672 raft_consensus.cc:348] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:04:44.655707 9668 raft_consensus.cc:348] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:04:44.656034 9672 raft_consensus.cc:374] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:44.656034 9668 raft_consensus.cc:374] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:44.656097 9668 raft_consensus.cc:729] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60cc09773b3e4a2496083b02d0f15f37, State: Initialized, Role: FOLLOWER
I20260810 23:04:44.656105 9672 raft_consensus.cc:729] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a15d42d77d4c149d8e8ea4192b6d73, State: Initialized, Role: FOLLOWER
I20260810 23:04:44.656620 9672 consensus_queue.cc:260] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:04:44.656625 9668 consensus_queue.cc:260] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:04:44.657392 9649 heartbeater.cc:503] Master 127.15.56.62:33657 was elected leader, sending a full tablet report...
I20260810 23:04:44.657748 9672 ts_tablet_manager.cc:1428] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:04:44.657766 9668 ts_tablet_manager.cc:1428] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:04:44.658056 9449 heartbeater.cc:503] Master 127.15.56.62:33657 was elected leader, sending a full tablet report...
W20260810 23:04:44.686332 9255 tablet.cc:2369] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:04:44.750975 9264 raft_consensus.cc:482] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:04:44.751235 9264 raft_consensus.cc:504] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:04:44.752434 9264 leader_election.cc:288] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465), 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631)
I20260810 23:04:44.756503 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.752781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:54967 (local address 127.15.56.2:44465)
0810 23:04:44.752944 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.752949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.753078 (+ 129us) server_negotiation.cc:407] Connection header received
0810 23:04:44.753257 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.753259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.753302 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.753367 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:44.753707 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.754290 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.755441 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.755896 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.755991 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.756056 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:44.756297 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 23:04:44.756368 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 23:04:44.756506 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.752865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:60923 (local address 127.15.56.3:40631)
0810 23:04:44.753051 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.753055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.753160 (+ 105us) server_negotiation.cc:407] Connection header received
0810 23:04:44.753291 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.753294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.753337 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.753412 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:44.753740 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.754307 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.755456 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.755907 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.756002 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.756063 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:44.756295 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 23:04:44.756373 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 23:04:44.756529 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.752636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:44465 (local address 127.15.56.1:54967)
0810 23:04:44.752995 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:44.753007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:44.753133 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:44.753385 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:44.753388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:44.753404 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:44.753600 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.753607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.754403 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.754407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.755318 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.755328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.756010 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.756014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.756134 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.756201 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 23:04:44.756266 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:04:44.756368 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":138,"spinlock_wait_cycles":37120,"thread_start_us":78,"threads_started":1}
I20260810 23:04:44.756551 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.752717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:40631 (local address 127.15.56.1:60923)
0810 23:04:44.753073 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:44.753086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:44.753192 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:44.753423 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:44.753426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:44.753468 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:44.753610 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.753615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.754415 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.754418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.755330 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:44.755340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.756020 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.756024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:44.756166 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.756196 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:04:44.756264 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:04:44.756389 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":130,"spinlock_wait_cycles":1792,"thread_start_us":101,"threads_started":1}
I20260810 23:04:44.757213 9385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f34fa85e141a41849646bc46bb61893c" candidate_uuid: "5c4239483857493fad13d728aa0e9cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" is_pre_election: true
I20260810 23:04:44.757413 9580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f34fa85e141a41849646bc46bb61893c" candidate_uuid: "5c4239483857493fad13d728aa0e9cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60cc09773b3e4a2496083b02d0f15f37" is_pre_election: true
I20260810 23:04:44.757580 9385 raft_consensus.cc:2454] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c4239483857493fad13d728aa0e9cdd in term 0.
I20260810 23:04:44.757681 9580 raft_consensus.cc:2454] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c4239483857493fad13d728aa0e9cdd in term 0.
I20260810 23:04:44.758116 9069 leader_election.cc:302] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c4239483857493fad13d728aa0e9cdd, 60cc09773b3e4a2496083b02d0f15f37; no voters:
I20260810 23:04:44.758424 9264 raft_consensus.cc:2781] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:04:44.758507 9264 raft_consensus.cc:482] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:04:44.758560 9264 raft_consensus.cc:3037] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:44.759614 9264 raft_consensus.cc:504] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:04:44.759984 9264 leader_election.cc:288] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 1 election: Requested vote from peers f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465), 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631)
I20260810 23:04:44.760366 9580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f34fa85e141a41849646bc46bb61893c" candidate_uuid: "5c4239483857493fad13d728aa0e9cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60cc09773b3e4a2496083b02d0f15f37"
I20260810 23:04:44.760366 9385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f34fa85e141a41849646bc46bb61893c" candidate_uuid: "5c4239483857493fad13d728aa0e9cdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73"
I20260810 23:04:44.760499 9580 raft_consensus.cc:3037] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:44.760499 9385 raft_consensus.cc:3037] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:44.761839 9385 raft_consensus.cc:2454] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c4239483857493fad13d728aa0e9cdd in term 1.
I20260810 23:04:44.761839 9580 raft_consensus.cc:2454] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c4239483857493fad13d728aa0e9cdd in term 1.
I20260810 23:04:44.762185 9069 leader_election.cc:302] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c4239483857493fad13d728aa0e9cdd, 60cc09773b3e4a2496083b02d0f15f37; no voters:
I20260810 23:04:44.762374 9264 raft_consensus.cc:2781] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:44.762606 9264 raft_consensus.cc:686] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 LEADER]: Becoming Leader. State: Replica: 5c4239483857493fad13d728aa0e9cdd, State: Running, Role: LEADER
I20260810 23:04:44.762907 9264 consensus_queue.cc:237] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:04:44.765947 8946 catalog_manager.cc:5495] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c4239483857493fad13d728aa0e9cdd (127.15.56.1). New cstate: current_term: 1 leader_uuid: "5c4239483857493fad13d728aa0e9cdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } health_report { overall_health: HEALTHY } } }
I20260810 23:04:44.788498 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.784768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.15.56.1:35907)
0810 23:04:44.785084 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:44.785091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:44.785103 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:04:44.785181 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:44.785184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:44.785241 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:44.785354 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:04:44.785600 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.786133 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.786979 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:44.787211 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:44.787300 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:44.787567 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:04:44.788075 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:04:44.788184 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:44.788277 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 23:04:44.788341 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260810 23:04:44.792824 9385 raft_consensus.cc:1261] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 1 FOLLOWER]: Refusing update from remote peer 5c4239483857493fad13d728aa0e9cdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:44.792824 9580 raft_consensus.cc:1261] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37 [term 1 FOLLOWER]: Refusing update from remote peer 5c4239483857493fad13d728aa0e9cdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:44.793547 9264 consensus_queue.cc:1035] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:44.793788 9684 consensus_queue.cc:1035] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:44.801676 9689 mvcc.cc:205] Tried to move back new op lower bound from 7317107035304488960 to 7317107035191721984. Current Snapshot: MvccSnapshot[applied={T|T < 7317107035304488960}]
I20260810 23:04:44.810407 8953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:44.810587 8953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:44.813155 8953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Sending DeleteTablet for 3 replicas of tablet f34fa85e141a41849646bc46bb61893c
I20260810 23:04:44.814005 9137 tablet_service.cc:1514] Processing DeleteTablet for tablet f34fa85e141a41849646bc46bb61893c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:44 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45442
I20260810 23:04:44.813997 9354 tablet_service.cc:1514] Processing DeleteTablet for tablet f34fa85e141a41849646bc46bb61893c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:44 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:43268
I20260810 23:04:44.814100 9551 tablet_service.cc:1514] Processing DeleteTablet for tablet f34fa85e141a41849646bc46bb61893c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:44 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:46380
I20260810 23:04:44.814410 9700 tablet_replica.cc:330] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: stopping tablet replica
I20260810 23:04:44.814546 9700 raft_consensus.cc:2229] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:04:44.814910 9702 tablet_replica.cc:330] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37: stopping tablet replica
I20260810 23:04:44.814908 9700 raft_consensus.cc:2258] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:44.814921 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 9014
I20260810 23:04:44.817713 9702 raft_consensus.cc:2229] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:44.818068 9702 raft_consensus.cc:2258] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:44.825147 9702 ts_tablet_manager.cc:1905] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:44.827755 9702 ts_tablet_manager.cc:1918] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:44.827858 9702 log.cc:1196] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/f34fa85e141a41849646bc46bb61893c
I20260810 23:04:44.828145 9702 ts_tablet_manager.cc:1939] T f34fa85e141a41849646bc46bb61893c P 60cc09773b3e4a2496083b02d0f15f37: Deleting consensus metadata
I20260810 23:04:44.828492 9701 tablet_replica.cc:330] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73: stopping tablet replica
I20260810 23:04:44.828850 9701 raft_consensus.cc:2229] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:44.829175 9701 raft_consensus.cc:2258] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:44.831194 8927 catalog_manager.cc:4841] TS 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631): tablet f34fa85e141a41849646bc46bb61893c (table test-table [id=2c2d04c9202742d9a5fb634ac76eb66c]) successfully deleted
I20260810 23:04:44.842054 9701 ts_tablet_manager.cc:1905] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:44.845058 9701 ts_tablet_manager.cc:1918] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:44.845173 9701 log.cc:1196] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/f34fa85e141a41849646bc46bb61893c
I20260810 23:04:44.845525 9701 ts_tablet_manager.cc:1939] T f34fa85e141a41849646bc46bb61893c P f5a15d42d77d4c149d8e8ea4192b6d73: Deleting consensus metadata
I20260810 23:04:44.846977 8929 catalog_manager.cc:4841] TS f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465): tablet f34fa85e141a41849646bc46bb61893c (table test-table [id=2c2d04c9202742d9a5fb634ac76eb66c]) successfully deleted
W20260810 23:04:44.848505 8930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:35907 (error 108)
I20260810 23:04:44.849351 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 9262
I20260810 23:04:44.849632 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:44.849346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:35907 (local address 127.0.0.1:45468)
0810 23:04:44.849482 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:44.849548 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:35907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:04:44.855639 8930 catalog_manager.cc:4584] TS 5c4239483857493fad13d728aa0e9cdd (127.15.56.1:35907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f34fa85e141a41849646bc46bb61893c: Network error: Client connection negotiation failed: client connection to 127.15.56.1:35907: connect: Connection refused (error 111)
I20260810 23:04:44.862563 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 9453
I20260810 23:04:44.880865 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 8900
I20260810 23:05:04.895040 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33657
--webserver_interface=127.15.56.62
--webserver_port=45279
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:33657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:05.013608 9704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:05.013899 9704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:05.013948 9704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:05.018046 9704 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:05:05.018106 9704 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:05:05.018121 9704 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:05:05.018133 9704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:05.018150 9704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:05.018174 9704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:05.018203 9704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:05.018229 9704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:05.023834 9704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:33657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:05.024992 9704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:05.026067 9704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:05:05.035082 9704 server_base.cc:1034] running on GCE node
W20260810 23:05:05.035185 9718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:05.035172 9715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:05.035158 9716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:05.036302 9704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:05.037484 9704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:05.038655 9704 hybrid_clock.cc:648] HybridClock initialized: now 1786403105038636 us; error 29 us; skew 500 ppm
Aug 10 23:05:05 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403105, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:05.046877 9704 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:05:05.048110 9704 webserver.cc:460] Webserver started at http://127.15.56.62:45279/ using document root <none> and password file <none>
I20260810 23:05:05.048771 9704 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:05.048853 9704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:05.053153 9704 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:05:05.055621 9726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:05.057013 9704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:05.057152 9704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "123e3d443f6946b4a052db96b8313ee9"
format_stamp: "Formatted at 2026-08-10 23:04:22 on dist-test-slave-6bbx"
server_key: "a6296df0d28b5953ffb8900d4db4bc42"
server_key_iv: "70b977ff14b6611bdce4f4a8dbdf9507"
server_key_version: "encryptionkey@0"
I20260810 23:05:05.057749 9704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:05.070626 9704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:05.073884 9704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:05.074072 9704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:05.082966 9704 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:33657
I20260810 23:05:05.083256 9806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:33657 every 8 connection(s)
I20260810 23:05:05.084172 9704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:05.088726 9807 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:05:05.090552 9807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Bootstrap starting.
I20260810 23:05:05.093430 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 9704
I20260810 23:05:05.093932 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:35907
--local_ip_for_outbound_sockets=127.15.56.1
--tserver_master_addrs=127.15.56.62:33657
--webserver_port=45271
--webserver_interface=127.15.56.1
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:05:05.109200 9807 log.cc:824] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:05.138029 9807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:05:05.138506 9807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Bootstrap complete.
I20260810 23:05:05.143390 9807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:05:05.143893 9807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123e3d443f6946b4a052db96b8313ee9, State: Initialized, Role: FOLLOWER
I20260810 23:05:05.144443 9807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:05:05.144570 9807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:05.144637 9807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:05.144738 9807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:05:05.146991 9807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:05:05.147396 9807 leader_election.cc:302] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 123e3d443f6946b4a052db96b8313ee9; no voters:
I20260810 23:05:05.147769 9807 leader_election.cc:288] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:05:05.147893 9811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:05:05.148080 9811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [term 3 LEADER]: Becoming Leader. State: Replica: 123e3d443f6946b4a052db96b8313ee9, State: Running, Role: LEADER
I20260810 23:05:05.148470 9811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } }
I20260810 23:05:05.149209 9807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:05.149572 9811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 123e3d443f6946b4a052db96b8313ee9. Latest consensus state: current_term: 3 leader_uuid: "123e3d443f6946b4a052db96b8313ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } } }
I20260810 23:05:05.149715 9811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:05.151335 9815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:05.151427 9812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "123e3d443f6946b4a052db96b8313ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123e3d443f6946b4a052db96b8313ee9" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33657 } } }
I20260810 23:05:05.151700 9812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:05.157343 9815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:05.158267 9815 catalog_manager.cc:1259] Loaded cluster ID: 296558d7dbb7434d854158f5673db1ba
I20260810 23:05:05.158349 9815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:05.161149 9815 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:05.162443 9815 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Loaded TSK: 0
I20260810 23:05:05.162804 9815 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 123e3d443f6946b4a052db96b8313ee9: Loaded TSK: 1
F20260810 23:05:05.163779 9815 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f67c21909ad google::LogMessage::Fail() at ??:0
@ 0x7f67c219388a google::LogMessage::SendToLog() at ??:0
@ 0x7f67c21904de google::LogMessage::Flush() at ??:0
@ 0x7f67c2191a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f67c4e5418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f67c4e4cf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f67ccaf57bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f67ccacf99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f67ccad0178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f67ccb11df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f67ccb3082b std::function<>::operator()() at ??:0
@ 0x7f67ccacfb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f67ccad0f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f67ccacd579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f67ccb11640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f67c43e23d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f67c43e2c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f67c43e43f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f67c43d3512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f67c5a176db start_thread at ??:0
@ 0x7f67c0c8e61f clone at ??:0
W20260810 23:05:05.262132 9809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:05.262590 9809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:05.262661 9809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:05.269050 9809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:05.269164 9809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:05.269260 9809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:05:05.277710 9809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:35907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=45271
--tserver_master_addrs=127.15.56.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:05.279266 9809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:05.280714 9809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:05.293277 9872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:05.293263 9873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:05.293612 9875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:05.293612 9809 server_base.cc:1034] running on GCE node
I20260810 23:05:05.294127 9809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:05.294894 9809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:05.296082 9809 hybrid_clock.cc:648] HybridClock initialized: now 1786403105296005 us; error 96 us; skew 500 ppm
Aug 10 23:05:05 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403105, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:05.299283 9809 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:05:05.300622 9809 webserver.cc:460] Webserver started at http://127.15.56.1:45271/ using document root <none> and password file <none>
I20260810 23:05:05.301451 9809 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:05.301563 9809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:05.307348 9809 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:05:05.310454 9884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:05.312276 9809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:05.312451 9809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "5c4239483857493fad13d728aa0e9cdd"
format_stamp: "Formatted at 2026-08-10 23:04:22 on dist-test-slave-6bbx"
server_key: "4b6a7c027b1283fcbd312401b15739a8"
server_key_iv: "fa68cfac543e83c6102bf9b41a3bc44e"
server_key_version: "encryptionkey@0"
I20260810 23:05:05.313268 9809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:05.337365 9809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:05.341665 9809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:05.341907 9809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:05.342728 9809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:05.344262 9894 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:05:05.349169 9809 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:05:05.349264 9809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 23:05:05.349350 9809 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:05:05.351863 9809 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:05:05.351944 9809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:05.352308 9894 tablet_bootstrap.cc:492] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: Bootstrap starting.
I20260810 23:05:05.363893 9809 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:35907
I20260810 23:05:05.363942 10052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:35907 every 8 connection(s)
I20260810 23:05:05.365288 9809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:05.366264 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 9809
I20260810 23:05:05.366811 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:44465
--local_ip_for_outbound_sockets=127.15.56.2
--tserver_master_addrs=127.15.56.62:33657
--webserver_port=39775
--webserver_interface=127.15.56.2
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:05:05.373853 9894 log.cc:824] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:05.381479 9894 tablet_bootstrap.cc:492] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:05:05.381884 9894 tablet_bootstrap.cc:492] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: Bootstrap complete.
I20260810 23:05:05.383033 9894 ts_tablet_manager.cc:1397] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.009s
I20260810 23:05:05.385637 9894 raft_consensus.cc:348] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:05:05.386116 9894 raft_consensus.cc:729] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4239483857493fad13d728aa0e9cdd, State: Initialized, Role: FOLLOWER
I20260810 23:05:05.386646 9894 consensus_queue.cc:260] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:05:05.387753 9894 ts_tablet_manager.cc:1428] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260810 23:05:05.448606 10059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:05:05.370147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:37899)
0810 23:05:05.371770 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:05.371846 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:05.372940 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:05.447785 (+ 74845us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1296,"thread_start_us":178,"threads_started":1}
W20260810 23:05:05.449246 10053 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:33657 (0 consecutive failures): Network error: Failed to ping master at 127.15.56.62:33657: Client connection negotiation failed: client connection to 127.15.56.62:33657: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:05:05.450081 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:05.449795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:41377)
0810 23:05:05.449945 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:05.450017 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 23:05:05.450850 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:05.450606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:34185)
0810 23:05:05.450735 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:05.450793 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 23:05:05.451057 10053 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:05:05.510092 10058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:05.510407 10058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:05.510454 10058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:05.514588 10058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:05.514667 10058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:05.514730 10058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:05:05.520457 10058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:44465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=39775
--tserver_master_addrs=127.15.56.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:05.521692 10058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:05.522863 10058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:05.533723 10075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:05.533767 10071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:05.533849 10058 server_base.cc:1034] running on GCE node
W20260810 23:05:05.533736 10079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:05.534487 10058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:05.535248 10058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:05.536449 10058 hybrid_clock.cc:648] HybridClock initialized: now 1786403105536412 us; error 50 us; skew 500 ppm
Aug 10 23:05:05 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403105, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:05.539546 10058 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:05:05.540544 10058 webserver.cc:460] Webserver started at http://127.15.56.2:39775/ using document root <none> and password file <none>
I20260810 23:05:05.541152 10058 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:05.541224 10058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:05.545284 10058 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:05:05.547631 10088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:05.549003 10058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:05.549124 10058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "f5a15d42d77d4c149d8e8ea4192b6d73"
format_stamp: "Formatted at 2026-08-10 23:04:23 on dist-test-slave-6bbx"
server_key: "3e2a34f440e8bcc57a43e9d7b45ed76c"
server_key_iv: "bd145948e275380ac38cdfb5b8eedce6"
server_key_version: "encryptionkey@0"
I20260810 23:05:05.549702 10058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:05.580166 10058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:05.583380 10058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:05.583578 10058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:05.584246 10058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:05.585107 10058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:05.585156 10058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:05.585212 10058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:05.585239 10058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:05.594636 10058 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:44465
I20260810 23:05:05.594666 10255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:44465 every 8 connection(s)
I20260810 23:05:05.595856 10058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:05.596719 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 10058
I20260810 23:05:05.597118 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:40631
--local_ip_for_outbound_sockets=127.15.56.3
--tserver_master_addrs=127.15.56.62:33657
--webserver_port=40237
--webserver_interface=127.15.56.3
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:05:05.601238 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:05.599256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:58561)
0810 23:05:05.599833 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:05.600453 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":116,"threads_started":1}
W20260810 23:05:05.601886 10256 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:33657 (0 consecutive failures): Network error: Failed to ping master at 127.15.56.62:33657: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
I20260810 23:05:05.602622 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:05.602373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:54555)
0810 23:05:05.602503 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:05.602557 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 23:05:05.603288 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:05.603079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:46123)
0810 23:05:05.603188 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:05.603235 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:05:05.603502 10256 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:05:05.715175 10258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:05.715468 10258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:05.715512 10258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:05.719900 10258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:05.719977 10258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:05.720041 10258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:05:05.725492 10258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:40631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=40237
--tserver_master_addrs=127.15.56.62:33657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:05.726598 10258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:05.727634 10258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:05.738119 10274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:05.738304 10258 server_base.cc:1034] running on GCE node
W20260810 23:05:05.738205 10272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:05.738123 10271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:05.738863 10258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:05.739611 10258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:05.740835 10258 hybrid_clock.cc:648] HybridClock initialized: now 1786403105740821 us; error 38 us; skew 500 ppm
Aug 10 23:05:05 dist-test-slave-6bbx krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403105, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:05.743861 10258 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:05:05.744948 10258 webserver.cc:460] Webserver started at http://127.15.56.3:40237/ using document root <none> and password file <none>
I20260810 23:05:05.745570 10258 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:05.745647 10258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:05.750152 10258 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 23:05:05.752652 10282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:05.754069 10258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:05:05.754204 10258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "60cc09773b3e4a2496083b02d0f15f37"
format_stamp: "Formatted at 2026-08-10 23:04:23 on dist-test-slave-6bbx"
server_key: "4b1f458945b7f77782a681eab060b4cd"
server_key_iv: "ff5bae343a3d2c81a74430f31d438f97"
server_key_version: "encryptionkey@0"
I20260810 23:05:05.754837 10258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:05.769802 10258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:05.773108 10258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:05.773340 10258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:05.774006 10258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:05.774915 10258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:05.774978 10258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:05.775027 10258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:05.775048 10258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:05.784397 10258 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:40631
I20260810 23:05:05.784428 10448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:40631 every 8 connection(s)
I20260810 23:05:05.785558 10258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:05:05.789599 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:05.787849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:46153)
0810 23:05:05.788321 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:05.788840 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
W20260810 23:05:05.790078 10449 heartbeater.cc:644] Failed to heartbeat to 127.15.56.62:33657 (0 consecutive failures): Network error: Failed to ping master at 127.15.56.62:33657: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
I20260810 23:05:05.790702 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:05.790459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:53211)
0810 23:05:05.790578 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:05.790629 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 23:05:05.791442 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:05.791209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:60649)
0810 23:05:05.791338 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:05.791388 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 23:05:05.791638 10449 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:05:05.795473 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 10258
I20260810 23:05:06.452203 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:06.451729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:57069)
0810 23:05:06.452010 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:06.452069 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260810 23:05:06.604650 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:06.604164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:54489)
0810 23:05:06.604468 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:06.604534 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
I20260810 23:05:06.792981 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:06.792471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:55359)
0810 23:05:06.792780 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:06.792854 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":87,"threads_started":1}
I20260810 23:05:06.896191 10458 raft_consensus.cc:482] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:06.896411 10458 raft_consensus.cc:504] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:05:06.897550 10458 leader_election.cc:288] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465), 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631)
Aug 10 23:05:06 dist-test-slave-6bbx krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403105, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.3@KRBTEST.COM
Aug 10 23:05:06 dist-test-slave-6bbx krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403105, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.2@KRBTEST.COM
I20260810 23:05:06.912650 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:06.897986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:52447 (local address 127.15.56.2:44465)
0810 23:05:06.898320 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:06.898344 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:06.898380 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:05:06.899085 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:06.899091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:06.899440 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:06.899718 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:06.901217 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:06.902242 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:06.904101 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:06.904615 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:06.904793 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:06.907960 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:06.907965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:06.907977 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:06.908011 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:06.909795 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:06.910472 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:06.910479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:06.910483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:06.910610 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:06.910998 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:06.911001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:06.911002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:06.911439 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:06.911564 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:06.912183 (+ 619us) server_negotiation.cc:299] Negotiation successful
0810 23:05:06.912454 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260810 23:05:06.912636 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:06.897804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:40631 (local address 127.15.56.1:37099)
0810 23:05:06.898136 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:06.898150 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:06.898291 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:06.899841 (+ 1550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:06.899848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:06.900422 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:06.901016 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:06.901035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:06.902413 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:06.902415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:06.903932 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:06.903945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:06.904719 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:06.904723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:06.904980 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:06.905759 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:06.905776 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:06.907460 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:06.909894 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:06.909901 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:06.909918 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:06.910349 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:06.910708 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:06.910711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:06.910713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:06.910884 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:06.911559 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:06.911565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:06.911855 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 23:05:06.912033 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 23:05:06.912439 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":179,"spinlock_wait_cycles":74240,"thread_start_us":100,"threads_started":1}
I20260810 23:05:06.912671 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:06.897997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:37099 (local address 127.15.56.3:40631)
0810 23:05:06.898407 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:06.898427 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:06.898459 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:05:06.899089 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:06.899094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:06.899440 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:06.899754 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:06.901221 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:06.902242 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:06.904071 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:06.904580 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:06.904730 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:06.907787 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:06.907796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:06.907828 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:06.907868 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:06.909753 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:06.910470 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:06.910476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:06.910481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:06.910610 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:06.910979 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:06.910983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:06.910985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:06.911440 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:06.911564 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:06.912180 (+ 616us) server_negotiation.cc:299] Negotiation successful
0810 23:05:06.912476 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":77,"threads_started":1}
I20260810 23:05:06.912678 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:06.897762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:44465 (local address 127.15.56.1:52447)
0810 23:05:06.897951 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:06.897966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:06.898076 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:06.899841 (+ 1765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:06.899848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:06.900422 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:06.901019 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:06.901035 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:06.902418 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:06.902421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:06.903932 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:06.903945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:06.904768 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:06.904772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:06.904986 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:06.905719 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:06.905753 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:06.907736 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:06.909928 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:06.909930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:06.909932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:06.910335 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:06.910707 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:06.910710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:06.910712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:06.910891 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:06.911558 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:06.911565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:06.911856 (+ 291us) client_negotiation.cc:769] Sending connection context
0810 23:05:06.912032 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 23:05:06.912518 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":570,"spinlock_wait_cycles":29824}
W20260810 23:05:06.915846 9885 leader_election.cc:341] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631): Not found: Tablet not found: f34fa85e141a41849646bc46bb61893c
W20260810 23:05:06.916356 9887 leader_election.cc:341] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465): Not found: Tablet not found: f34fa85e141a41849646bc46bb61893c
I20260810 23:05:06.916456 9887 leader_election.cc:302] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c4239483857493fad13d728aa0e9cdd; no voters: 60cc09773b3e4a2496083b02d0f15f37, f5a15d42d77d4c149d8e8ea4192b6d73
I20260810 23:05:06.916747 10458 raft_consensus.cc:2726] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:05:07.453488 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:07.452980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:41305)
0810 23:05:07.453294 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:07.453360 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
I20260810 23:05:07.606010 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:07.605550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:42429)
0810 23:05:07.605841 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:07.605905 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260810 23:05:07.794332 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:07.793869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:52669)
0810 23:05:07.794165 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:07.794228 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
I20260810 23:05:08.454758 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:08.454279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:47187)
0810 23:05:08.454572 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:08.454636 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":98,"threads_started":1}
I20260810 23:05:08.607446 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:08.606933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:36683)
0810 23:05:08.607229 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:08.607299 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
I20260810 23:05:08.795791 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:08.795260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:33111)
0810 23:05:08.795567 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:08.795637 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
I20260810 23:05:09.033828 10472 raft_consensus.cc:482] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:09.033983 10472 raft_consensus.cc:504] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:05:09.034440 10472 leader_election.cc:288] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465), 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631)
W20260810 23:05:09.035372 9885 leader_election.cc:341] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631): Not found: Tablet not found: f34fa85e141a41849646bc46bb61893c
W20260810 23:05:09.035491 9887 leader_election.cc:341] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465): Not found: Tablet not found: f34fa85e141a41849646bc46bb61893c
I20260810 23:05:09.035588 9887 leader_election.cc:302] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c4239483857493fad13d728aa0e9cdd; no voters: 60cc09773b3e4a2496083b02d0f15f37, f5a15d42d77d4c149d8e8ea4192b6d73
I20260810 23:05:09.035773 10472 raft_consensus.cc:2726] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:05:09.456205 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:09.455606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:51781)
0810 23:05:09.455995 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:09.456072 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":117,"threads_started":1}
I20260810 23:05:09.608906 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:09.608438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:50245)
0810 23:05:09.608723 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:09.608792 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
I20260810 23:05:09.797041 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:09.796596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:49893)
0810 23:05:09.796880 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:09.796941 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260810 23:05:10.457643 10479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:10.457183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:43643)
0810 23:05:10.457481 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:10.457541 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260810 23:05:10.610250 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:10.609763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:38037)
0810 23:05:10.610052 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:10.610119 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260810 23:05:10.798394 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:10.797914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:47239)
0810 23:05:10.798221 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:10.798288 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20260810 23:05:11.459043 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:11.458515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:40807)
0810 23:05:11.458839 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:11.458931 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":105,"threads_started":1}
I20260810 23:05:11.611595 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:11.611108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:40027)
0810 23:05:11.611429 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:11.611495 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
I20260810 23:05:11.799862 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:11.799338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:48361)
0810 23:05:11.799653 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:11.799756 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":88,"threads_started":1}
I20260810 23:05:11.848114 10489 raft_consensus.cc:482] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:11.848284 10489 raft_consensus.cc:504] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:05:11.848712 10489 leader_election.cc:288] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465), 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631)
W20260810 23:05:11.849546 9887 leader_election.cc:341] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465): Not found: Tablet not found: f34fa85e141a41849646bc46bb61893c
W20260810 23:05:11.849664 9885 leader_election.cc:341] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631): Not found: Tablet not found: f34fa85e141a41849646bc46bb61893c
I20260810 23:05:11.849834 9885 leader_election.cc:302] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c4239483857493fad13d728aa0e9cdd; no voters: 60cc09773b3e4a2496083b02d0f15f37, f5a15d42d77d4c149d8e8ea4192b6d73
I20260810 23:05:11.849982 10489 raft_consensus.cc:2726] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:05:12.460487 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:12.460005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:44583)
0810 23:05:12.460312 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:12.460378 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":78,"threads_started":1}
I20260810 23:05:12.613112 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:12.612560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:51033)
0810 23:05:12.612953 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:12.613012 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":161,"threads_started":1}
I20260810 23:05:12.801225 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:12.800693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:49403)
0810 23:05:12.801050 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:12.801107 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260810 23:05:13.461840 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:13.461386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:50467)
0810 23:05:13.461662 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:13.461736 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260810 23:05:13.614389 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:13.613916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:47381)
0810 23:05:13.614221 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:13.614287 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260810 23:05:13.802613 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:13.802079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:33187)
0810 23:05:13.802439 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:13.802504 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":117,"threads_started":1}
I20260810 23:05:14.463286 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:14.462803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:42347)
0810 23:05:14.463106 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:14.463176 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260810 23:05:14.615916 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:14.615395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:43159)
0810 23:05:14.615744 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:14.615814 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
I20260810 23:05:14.804164 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:14.803599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:46673)
0810 23:05:14.803980 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:14.804059 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":126,"threads_started":1}
I20260810 23:05:15.388255 10509 raft_consensus.cc:482] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:15.388406 10509 raft_consensus.cc:504] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a15d42d77d4c149d8e8ea4192b6d73" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 44465 } } peers { permanent_uuid: "60cc09773b3e4a2496083b02d0f15f37" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 40631 } } peers { permanent_uuid: "5c4239483857493fad13d728aa0e9cdd" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 35907 } }
I20260810 23:05:15.388839 10509 leader_election.cc:288] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465), 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631)
W20260810 23:05:15.389660 9887 leader_election.cc:341] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f5a15d42d77d4c149d8e8ea4192b6d73 (127.15.56.2:44465): Not found: Tablet not found: f34fa85e141a41849646bc46bb61893c
W20260810 23:05:15.389791 9885 leader_election.cc:341] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 60cc09773b3e4a2496083b02d0f15f37 (127.15.56.3:40631): Not found: Tablet not found: f34fa85e141a41849646bc46bb61893c
I20260810 23:05:15.389900 9885 leader_election.cc:302] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c4239483857493fad13d728aa0e9cdd; no voters: 60cc09773b3e4a2496083b02d0f15f37, f5a15d42d77d4c149d8e8ea4192b6d73
I20260810 23:05:15.390041 10509 raft_consensus.cc:2726] T f34fa85e141a41849646bc46bb61893c P 5c4239483857493fad13d728aa0e9cdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:05:15.464672 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:15.464216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:33267)
0810 23:05:15.464489 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:15.464552 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260810 23:05:15.617342 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:15.616836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:54329)
0810 23:05:15.617136 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:15.617226 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":102,"threads_started":1}
I20260810 23:05:15.805672 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:15.805152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:35011)
0810 23:05:15.805489 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:15.805564 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":91,"threads_started":1}
I20260810 23:05:16.465970 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:16.465523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:50367)
0810 23:05:16.465802 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:16.465866 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260810 23:05:16.618623 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:16.618106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:55975)
0810 23:05:16.618443 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:16.618502 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":123,"threads_started":1}
I20260810 23:05:16.806962 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:16.806492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:54491)
0810 23:05:16.806796 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:16.806858 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":98,"threads_started":1}
I20260810 23:05:17.467407 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:17.466877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:38663)
0810 23:05:17.467222 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:17.467297 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":91,"threads_started":1}
I20260810 23:05:17.620033 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:17.619500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:59649)
0810 23:05:17.619850 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:17.619921 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":103,"threads_started":1}
I20260810 23:05:17.808368 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:17.807916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:50675)
0810 23:05:17.808196 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:17.808260 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260810 23:05:18.468921 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:18.468396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:35487)
0810 23:05:18.468743 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:18.468813 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":86,"threads_started":1}
I20260810 23:05:18.621342 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:18.620878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:48143)
0810 23:05:18.621150 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:18.621216 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260810 23:05:18.809700 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:18.809225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:34829)
0810 23:05:18.809523 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:18.809592 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
I20260810 23:05:19.470448 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:19.469939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:38059)
0810 23:05:19.470245 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:19.470339 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":85,"threads_started":1}
I20260810 23:05:19.622720 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:19.622160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:50917)
0810 23:05:19.622524 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:19.622596 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":80,"threads_started":1}
I20260810 23:05:19.811111 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:19.810611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:57579)
0810 23:05:19.810932 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:19.811004 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
I20260810 23:05:20.472172 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:20.471640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.1:54845)
0810 23:05:20.472000 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:20.472067 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":82,"threads_started":1}
I20260810 23:05:20.624145 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:20.623730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.2:54529)
0810 23:05:20.623979 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:20.624041 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260810 23:05:20.798748 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 9809
I20260810 23:05:20.812705 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:20.812233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33657 (local address 127.15.56.3:37099)
0810 23:05:20.812541 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:20.812604 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.62:33657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":100,"threads_started":1}
I20260810 23:05:20.814495 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 10058
I20260810 23:05:20.836577 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 10258
2026-08-10T23:05:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59754 ms)
[----------] 30 tests from SecurityITest (253205 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:20 dist-test-slave-6bbx krb5kdc[10536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:20 dist-test-slave-6bbx krb5kdc[10536](info): set up 2 sockets
Aug 10 23:05:20 dist-test-slave-6bbx krb5kdc[10536](info): commencing operation
krb5kdc: starting...
W20260810 23:05:22.014863 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:22 dist-test-slave-6bbx krb5kdc[10536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:05:22.154696 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:41031
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:41031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:22.274380 10553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:22.274714 10553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:22.274760 10553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:22.278972 10553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:22.279047 10553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:22.279067 10553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:22.279083 10553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:22.279098 10553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:22.284711 10553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:41031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:41031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:22.285853 10553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:22.286899 10553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:22.296438 10560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:22.296514 10562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:22.296506 10553 server_base.cc:1034] running on GCE node
W20260810 23:05:22.296439 10559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:22.297298 10553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:22.298545 10553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:22.299752 10553 hybrid_clock.cc:648] HybridClock initialized: now 1786403122299719 us; error 47 us; skew 500 ppm
Aug 10 23:05:22 dist-test-slave-6bbx krb5kdc[10536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403122, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:22.302838 10553 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:05:22.303923 10553 webserver.cc:460] Webserver started at http://127.15.56.62:34281/ using document root <none> and password file <none>
I20260810 23:05:22.304536 10553 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:22.304608 10553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:22.304836 10553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:22.307194 10553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "c51d3153b91145ee827cb7f22a6f28b3"
format_stamp: "Formatted at 2026-08-10 23:05:22 on dist-test-slave-6bbx"
server_key: "98267257e86003c833e9f7d181bab13f"
server_key_iv: "28d6210c6371bcca58c649082d6b692b"
server_key_version: "encryptionkey@0"
I20260810 23:05:22.307868 10553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "c51d3153b91145ee827cb7f22a6f28b3"
format_stamp: "Formatted at 2026-08-10 23:05:22 on dist-test-slave-6bbx"
server_key: "98267257e86003c833e9f7d181bab13f"
server_key_iv: "28d6210c6371bcca58c649082d6b692b"
server_key_version: "encryptionkey@0"
I20260810 23:05:22.312122 10553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:05:22.315040 10572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:22.316712 10553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:22.316920 10553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "c51d3153b91145ee827cb7f22a6f28b3"
format_stamp: "Formatted at 2026-08-10 23:05:22 on dist-test-slave-6bbx"
server_key: "98267257e86003c833e9f7d181bab13f"
server_key_iv: "28d6210c6371bcca58c649082d6b692b"
server_key_version: "encryptionkey@0"
I20260810 23:05:22.317062 10553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:22.353773 10553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:22.357595 10553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:22.357815 10553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:22.365566 10553 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:41031
I20260810 23:05:22.365622 10639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:41031 every 8 connection(s)
I20260810 23:05:22.366657 10553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:22.369801 10640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:22.373032 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 10553
I20260810 23:05:22.373159 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:22.373433 15584 external_mini_cluster.cc:1428] Setting key b20c587dc24a29e219c3ddfbab909b15
I20260810 23:05:22.376593 10640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3: Bootstrap starting.
I20260810 23:05:22.379135 10640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:22.380034 10640 log.cc:824] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:22.382226 10640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3: No bootstrap required, opened a new log
Aug 10 23:05:22 dist-test-slave-6bbx krb5kdc[10536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:05:22.385967 10640 raft_consensus.cc:348] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51d3153b91145ee827cb7f22a6f28b3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41031 } }
I20260810 23:05:22.386262 10640 raft_consensus.cc:374] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:22.386327 10640 raft_consensus.cc:729] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c51d3153b91145ee827cb7f22a6f28b3, State: Initialized, Role: FOLLOWER
I20260810 23:05:22.386857 10640 consensus_queue.cc:260] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51d3153b91145ee827cb7f22a6f28b3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41031 } }
I20260810 23:05:22.387028 10640 raft_consensus.cc:388] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:22.387101 10640 raft_consensus.cc:482] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:22.387219 10640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:22.388335 10640 raft_consensus.cc:504] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51d3153b91145ee827cb7f22a6f28b3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41031 } }
I20260810 23:05:22.388753 10640 leader_election.cc:302] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c51d3153b91145ee827cb7f22a6f28b3; no voters:
I20260810 23:05:22.389117 10640 leader_election.cc:288] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:22.389238 10645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:22.389472 10645 raft_consensus.cc:686] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [term 1 LEADER]: Becoming Leader. State: Replica: c51d3153b91145ee827cb7f22a6f28b3, State: Running, Role: LEADER
I20260810 23:05:22.389530 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:22.374967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.15.56.62:41031)
0810 23:05:22.375623 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:22.375643 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:22.375708 (+ 65us) server_negotiation.cc:407] Connection header received
0810 23:05:22.376437 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:22.376457 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:22.376848 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:22.377192 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:22.378043 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:22.379163 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:22.380126 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:22.380666 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:22.380829 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:22.383398 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:22.383404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:22.383421 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:22.383453 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:22.386037 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:22.386605 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:22.386610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:22.386616 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:22.386750 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:22.387046 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:22.387049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:22.387051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:22.387585 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:22.387831 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:22.388150 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 23:05:22.388514 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":160,"threads_started":1}
I20260810 23:05:22.389932 10645 consensus_queue.cc:237] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51d3153b91145ee827cb7f22a6f28b3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41031 } }
I20260810 23:05:22.390532 10640 sys_catalog.cc:564] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:22.391268 10645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c51d3153b91145ee827cb7f22a6f28b3. Latest consensus state: current_term: 1 leader_uuid: "c51d3153b91145ee827cb7f22a6f28b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51d3153b91145ee827cb7f22a6f28b3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41031 } } }
I20260810 23:05:22.391389 10645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:22.392851 10646 sys_catalog.cc:455] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c51d3153b91145ee827cb7f22a6f28b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c51d3153b91145ee827cb7f22a6f28b3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 41031 } } }
I20260810 23:05:22.393007 10646 sys_catalog.cc:458] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3 [sys.catalog]: This master's current role is: LEADER
W20260810 23:05:22.395864 10672 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:05:22.395972 10672 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:22.396101 10658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:22.396867 10658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:22.403573 10658 catalog_manager.cc:1347] Generated new cluster ID: cc3593aeab9f49c29aa061ccb93c6e67
I20260810 23:05:22.403710 10658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:22.409510 10658 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:22.410969 10658 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:22.420938 10658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3: Generated new TSK 0
I20260810 23:05:22.422008 10658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:05:22.461052 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41031
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:22.584170 10681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:22.584512 10681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:22.584559 10681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:22.589299 10681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:22.589402 10681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:22.589470 10681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:05:22.596258 10681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:22.597504 10681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:22.598621 10681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:22.609683 10688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:22.609687 10687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:22.609687 10691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:22.610210 10681 server_base.cc:1034] running on GCE node
I20260810 23:05:22.610740 10681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:22.611481 10681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:22.612704 10681 hybrid_clock.cc:648] HybridClock initialized: now 1786403122612667 us; error 55 us; skew 500 ppm
Aug 10 23:05:22 dist-test-slave-6bbx krb5kdc[10536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403122, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:22.616067 10681 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:05:22.617107 10681 webserver.cc:460] Webserver started at http://127.15.56.1:45139/ using document root <none> and password file <none>
I20260810 23:05:22.617722 10681 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:22.617797 10681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:22.618009 10681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:22.620304 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "d231d9bebdfd4f7bb14c08eb96782756"
format_stamp: "Formatted at 2026-08-10 23:05:22 on dist-test-slave-6bbx"
server_key: "d88e6b3585ea9e62010ab9660592cac9"
server_key_iv: "2833b78304d5e8bc94db12142315ca40"
server_key_version: "encryptionkey@0"
I20260810 23:05:22.620990 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "d231d9bebdfd4f7bb14c08eb96782756"
format_stamp: "Formatted at 2026-08-10 23:05:22 on dist-test-slave-6bbx"
server_key: "d88e6b3585ea9e62010ab9660592cac9"
server_key_iv: "2833b78304d5e8bc94db12142315ca40"
server_key_version: "encryptionkey@0"
I20260810 23:05:22.625234 10681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:22.627859 10702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:22.629171 10681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:22.629305 10681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "d231d9bebdfd4f7bb14c08eb96782756"
format_stamp: "Formatted at 2026-08-10 23:05:22 on dist-test-slave-6bbx"
server_key: "d88e6b3585ea9e62010ab9660592cac9"
server_key_iv: "2833b78304d5e8bc94db12142315ca40"
server_key_version: "encryptionkey@0"
I20260810 23:05:22.629412 10681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:22.658443 10681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:22.662264 10681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:22.662475 10681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:22.663133 10681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:22.664070 10681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:22.664124 10681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:22.664181 10681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:22.664211 10681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:22.673389 10681 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:37171
I20260810 23:05:22.673436 10865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:37171 every 8 connection(s)
I20260810 23:05:22.674464 10681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:22.680022 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 10681
I20260810 23:05:22.680166 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:05:22.680447 15584 external_mini_cluster.cc:1428] Setting key f2a4411fafc0b4482b20934c2fb8e0e3
Aug 10 23:05:22 dist-test-slave-6bbx krb5kdc[10536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403122, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:05:22.690443 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:22.676634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:44147 (local address 127.15.56.62:41031)
0810 23:05:22.676786 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:22.676790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:22.677787 (+ 997us) server_negotiation.cc:407] Connection header received
0810 23:05:22.679277 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:22.679281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:22.679343 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:22.679431 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:22.680922 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:22.681637 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:22.682923 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:22.683162 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:22.683295 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:22.686208 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:22.686213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:22.686215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:22.686249 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:22.688011 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:22.688621 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:22.688625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:22.688627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:22.688782 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:22.689161 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:22.689165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:22.689167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:22.689433 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:22.689539 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:22.690098 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 23:05:22.690261 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:05:22.691355 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:22.676996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41031 (local address 127.15.56.1:44147)
0810 23:05:22.677570 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:22.677626 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:22.678944 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:22.679616 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:22.679624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:22.680128 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:22.680755 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:22.680774 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:22.681806 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:22.681810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:22.682786 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:22.682797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:22.683331 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:22.683335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:22.683531 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:22.684198 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:22.684217 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:22.685975 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:22.688150 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:22.688157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:22.688175 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:22.688512 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:22.688894 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:22.688896 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:22.688897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:22.689063 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:22.689532 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:22.689538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:22.689838 (+ 300us) client_negotiation.cc:769] Sending connection context
0810 23:05:22.690015 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 23:05:22.690331 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":177,"threads_started":1}
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:05:22.693207 10866 heartbeater.cc:347] Connected to a master server at 127.15.56.62:41031
I20260810 23:05:22.693506 10866 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:22.694190 10866 heartbeater.cc:511] Master 127.15.56.62:41031 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
I20260810 23:05:22.696175 10596 ts_manager.cc:194] Registered new tserver with Master: d231d9bebdfd4f7bb14c08eb96782756 (127.15.56.1:37171)
I20260810 23:05:22.697399 10596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:44147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:05:22.715169 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41031
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:22.834980 10877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:22.835279 10877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:22.835325 10877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:22.839468 10877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:22.839555 10877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:22.839615 10877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:05:22.845305 10877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:22.846482 10877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:22.847581 10877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:22.857878 10883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:22.858065 10877 server_base.cc:1034] running on GCE node
W20260810 23:05:22.857883 10887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:22.858249 10884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:22.858656 10877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:22.859354 10877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:22.860514 10877 hybrid_clock.cc:648] HybridClock initialized: now 1786403122860494 us; error 31 us; skew 500 ppm
Aug 10 23:05:22 dist-test-slave-6bbx krb5kdc[10536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403122, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:22.863354 10877 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:05:22.864341 10877 webserver.cc:460] Webserver started at http://127.15.56.2:39383/ using document root <none> and password file <none>
I20260810 23:05:22.864944 10877 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:22.865016 10877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:22.865231 10877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:22.867348 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "43e29730fe814e838874a19ff2763393"
format_stamp: "Formatted at 2026-08-10 23:05:22 on dist-test-slave-6bbx"
server_key: "6ded9d6f88c802323a47872d680d2724"
server_key_iv: "2c613a97cda3425c4d21e8597bd2b8de"
server_key_version: "encryptionkey@0"
I20260810 23:05:22.867914 10877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "43e29730fe814e838874a19ff2763393"
format_stamp: "Formatted at 2026-08-10 23:05:22 on dist-test-slave-6bbx"
server_key: "6ded9d6f88c802323a47872d680d2724"
server_key_iv: "2c613a97cda3425c4d21e8597bd2b8de"
server_key_version: "encryptionkey@0"
I20260810 23:05:22.871706 10877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:05:22.874267 10899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:22.875527 10877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:05:22.875639 10877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "43e29730fe814e838874a19ff2763393"
format_stamp: "Formatted at 2026-08-10 23:05:22 on dist-test-slave-6bbx"
server_key: "6ded9d6f88c802323a47872d680d2724"
server_key_iv: "2c613a97cda3425c4d21e8597bd2b8de"
server_key_version: "encryptionkey@0"
I20260810 23:05:22.875775 10877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:22.888409 10877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:22.891896 10877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:22.892086 10877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:22.892748 10877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:22.893713 10877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:22.893762 10877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:22.893805 10877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:22.893817 10877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:22.902649 10877 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:36959
I20260810 23:05:22.902675 11064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:36959 every 8 connection(s)
I20260810 23:05:22.903584 10877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:22.913115 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 10877
I20260810 23:05:22.913229 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:05:22.913553 15584 external_mini_cluster.cc:1428] Setting key 47c7b745a2e22818106dad0742270d0e
Aug 10 23:05:22 dist-test-slave-6bbx krb5kdc[10536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403122, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:05:22.918467 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:22.905606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:52855 (local address 127.15.56.62:41031)
0810 23:05:22.905790 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:22.905794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:22.906755 (+ 961us) server_negotiation.cc:407] Connection header received
0810 23:05:22.908150 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:22.908154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:22.908226 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:22.908299 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:22.909591 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:22.910126 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:22.911336 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:22.911576 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:22.911703 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:22.914638 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:22.914643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:22.914645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:22.914672 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:22.916292 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:22.916845 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:22.916849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:22.916851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:22.916934 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:22.917298 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:22.917301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:22.917303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:22.917499 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:22.917588 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:22.918159 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 23:05:22.918301 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 23:05:22.919433 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:22.906034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41031 (local address 127.15.56.2:52855)
0810 23:05:22.906554 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:22.906604 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:22.907930 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:22.908499 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:22.908507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:22.908911 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:22.909436 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:22.909452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:22.910256 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:22.910260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:22.911205 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:22.911217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:22.911780 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:22.911785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:22.911970 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:22.912604 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:22.912623 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:22.914455 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:22.916415 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:22.916422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:22.916442 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:22.916743 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:22.917022 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:22.917025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:22.917027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:22.917203 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:22.917602 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:22.917610 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:22.917905 (+ 295us) client_negotiation.cc:769] Sending connection context
0810 23:05:22.918096 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 23:05:22.918499 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":128,"threads_started":1}
I20260810 23:05:22.920843 11065 heartbeater.cc:347] Connected to a master server at 127.15.56.62:41031
I20260810 23:05:22.921152 11065 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:22.921823 11065 heartbeater.cc:511] Master 127.15.56.62:41031 requested a full tablet report, sending...
I20260810 23:05:22.923213 10596 ts_manager.cc:194] Registered new tserver with Master: 43e29730fe814e838874a19ff2763393 (127.15.56.2:36959)
I20260810 23:05:22.924086 10596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:52855
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:05:22.945160 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41031
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:23.064783 11073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:23.065099 11073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:23.065147 11073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:23.069437 11073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:23.069545 11073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:23.069638 11073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:05:23.077292 11073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:41031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:23.078953 11073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:23.080500 11073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:23.091545 11078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:23.091658 11073 server_base.cc:1034] running on GCE node
W20260810 23:05:23.091537 11080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:23.091537 11082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:23.092470 11073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:23.093235 11073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:23.094416 11073 hybrid_clock.cc:648] HybridClock initialized: now 1786403123094378 us; error 52 us; skew 500 ppm
Aug 10 23:05:23 dist-test-slave-6bbx krb5kdc[10536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403123, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:23.097656 11073 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:05:23.098835 11073 webserver.cc:460] Webserver started at http://127.15.56.3:37685/ using document root <none> and password file <none>
I20260810 23:05:23.099646 11073 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:23.099818 11073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:23.100072 11073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:23.102617 11073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "80e5619de4fd47578ec03c1e7925795b"
format_stamp: "Formatted at 2026-08-10 23:05:23 on dist-test-slave-6bbx"
server_key: "ea4a0744952d006d57c611f5db166126"
server_key_iv: "d1d5c9a5bd60cc6ff4b548b96f56b40c"
server_key_version: "encryptionkey@0"
I20260810 23:05:23.103282 11073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "80e5619de4fd47578ec03c1e7925795b"
format_stamp: "Formatted at 2026-08-10 23:05:23 on dist-test-slave-6bbx"
server_key: "ea4a0744952d006d57c611f5db166126"
server_key_iv: "d1d5c9a5bd60cc6ff4b548b96f56b40c"
server_key_version: "encryptionkey@0"
I20260810 23:05:23.107710 11073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260810 23:05:23.110723 11093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:23.112267 11073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 23:05:23.112411 11073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "80e5619de4fd47578ec03c1e7925795b"
format_stamp: "Formatted at 2026-08-10 23:05:23 on dist-test-slave-6bbx"
server_key: "ea4a0744952d006d57c611f5db166126"
server_key_iv: "d1d5c9a5bd60cc6ff4b548b96f56b40c"
server_key_version: "encryptionkey@0"
I20260810 23:05:23.112552 11073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:23.145624 11073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:23.148840 11073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:23.149008 11073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:23.149617 11073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:23.150476 11073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:23.150525 11073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:23.150568 11073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:23.150583 11073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:23.159750 11073 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:43041
I20260810 23:05:23.159830 11257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:43041 every 8 connection(s)
I20260810 23:05:23.160770 11073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:05:23.163501 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 11073
I20260810 23:05:23.163638 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:05:23.163944 15584 external_mini_cluster.cc:1428] Setting key c0602d6ebf072a477dec3bdff13c4b0c
Aug 10 23:05:23 dist-test-slave-6bbx krb5kdc[10536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403123, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:05:23.176077 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.162799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:45043 (local address 127.15.56.62:41031)
0810 23:05:23.162952 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:23.162956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:23.164328 (+ 1372us) server_negotiation.cc:407] Connection header received
0810 23:05:23.165348 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:23.165351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:23.165410 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:23.165487 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:23.166772 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.167318 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.168610 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.168844 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.168939 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.172233 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:23.172237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:23.172239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:23.172272 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:23.173745 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:23.174383 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:23.174386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:23.174387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:23.174473 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:23.174800 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:23.174802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:23.174803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:23.175009 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:23.175081 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:23.175809 (+ 728us) server_negotiation.cc:299] Negotiation successful
0810 23:05:23.175943 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:05:23.177131 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.163212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:41031 (local address 127.15.56.3:45043)
0810 23:05:23.164127 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:23.164199 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:23.165124 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:23.165685 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:23.165693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:23.166080 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:23.166620 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.166637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.167458 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.167461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.168473 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.168485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.169015 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.169020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.169217 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.169904 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:23.170004 (+ 100us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:23.171988 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:23.173905 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:23.173913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:23.173932 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:23.174265 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:23.174547 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:23.174549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:23.174550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:23.174722 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:23.175107 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:23.175114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:23.175431 (+ 317us) client_negotiation.cc:769] Sending connection context
0810 23:05:23.175636 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 23:05:23.176044 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":138,"threads_started":1}
I20260810 23:05:23.178419 11259 heartbeater.cc:347] Connected to a master server at 127.15.56.62:41031
I20260810 23:05:23.178678 11259 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:23.179248 11259 heartbeater.cc:511] Master 127.15.56.62:41031 requested a full tablet report, sending...
I20260810 23:05:23.180369 10596 ts_manager.cc:194] Registered new tserver with Master: 80e5619de4fd47578ec03c1e7925795b (127.15.56.3:43041)
I20260810 23:05:23.180975 10596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:45043
I20260810 23:05:23.190496 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:05:23.200942 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.192853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.15.56.62:41031)
0810 23:05:23.193041 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:23.193047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:23.193161 (+ 114us) server_negotiation.cc:407] Connection header received
0810 23:05:23.193308 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:23.193311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:23.193361 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:23.193432 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:23.194403 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.194961 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.195879 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.196101 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.196195 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.197705 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:23.197708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:23.197710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:23.197740 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:23.199142 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:23.199578 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:23.199581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:23.199583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:23.199727 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:23.199993 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:23.199997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:23.199998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:23.200253 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:23.200384 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:23.200628 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 23:05:23.200779 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 23:05:23.204232 10596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:05:23.207594 10596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:05:23.222821 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.217272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.15.56.2:36959)
0810 23:05:23.218044 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:23.218058 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:23.218084 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:05:23.218169 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:23.218175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:23.218408 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:23.218614 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:23.218941 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.219897 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.221319 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.222096 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.222216 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.222388 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:23.222556 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 23:05:23.222657 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":137,"threads_started":1}
I20260810 23:05:23.222815 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.216913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:36959 (local address 127.0.0.1:39766)
0810 23:05:23.217825 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:23.217881 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:23.218018 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:23.218614 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:23.218620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:23.218644 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:23.218806 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.218814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.220023 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.220027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.221197 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.221206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.222232 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.222236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.222346 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.222394 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 23:05:23.222514 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 23:05:23.222647 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260810 23:05:23.223453 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.217698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:43041 (local address 127.0.0.1:53688)
0810 23:05:23.218535 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:23.218547 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:23.218628 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:23.219097 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:23.219099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:23.219116 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:23.219199 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.219204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.220523 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.220526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.221677 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.221684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.223063 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.223067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.223153 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.223179 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:05:23.223257 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:05:23.223329 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":100,"threads_started":1}
I20260810 23:05:23.223558 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.218094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.15.56.3:43041)
0810 23:05:23.218476 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:23.218488 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:23.218725 (+ 237us) server_negotiation.cc:407] Connection header received
0810 23:05:23.218807 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:23.218812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:23.218979 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:23.219132 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:23.219290 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.220386 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.221795 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.222607 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.222707 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.222854 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:23.223314 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 23:05:23.223417 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":175,"threads_started":1}
I20260810 23:05:23.223978 11278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.217229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.15.56.1:37171)
0810 23:05:23.217829 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:23.217839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:23.218457 (+ 618us) server_negotiation.cc:407] Connection header received
0810 23:05:23.218543 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:23.218549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:23.218747 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:23.218889 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:23.219035 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.220233 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.221823 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.222644 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.222756 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.222892 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:23.223717 (+ 825us) server_negotiation.cc:299] Negotiation successful
0810 23:05:23.223825 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":110,"threads_started":1}
I20260810 23:05:23.223591 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.217024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:37171 (local address 127.0.0.1:39152)
0810 23:05:23.218259 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:23.218273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:23.218363 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:23.218858 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:23.218861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:23.218878 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:23.218957 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.218962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.220366 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.220369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.221656 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.221668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.223199 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.223203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.223310 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.223335 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:05:23.223405 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:05:23.223474 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"mutex_wait_us":6,"thread_start_us":118,"threads_started":1}
I20260810 23:05:23.226256 10967 tablet_service.cc:1467] Processing CreateTablet for tablet d749a04bf1164bc9bebfa99c777e7d24 (DEFAULT_TABLE table=test-table [id=9aabf365c78d49c9ad273e31a63252dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:23.226743 10771 tablet_service.cc:1467] Processing CreateTablet for tablet d749a04bf1164bc9bebfa99c777e7d24 (DEFAULT_TABLE table=test-table [id=9aabf365c78d49c9ad273e31a63252dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:23.226737 11166 tablet_service.cc:1467] Processing CreateTablet for tablet d749a04bf1164bc9bebfa99c777e7d24 (DEFAULT_TABLE table=test-table [id=9aabf365c78d49c9ad273e31a63252dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:23.227353 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d749a04bf1164bc9bebfa99c777e7d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:23.227926 11166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d749a04bf1164bc9bebfa99c777e7d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:23.227926 10771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d749a04bf1164bc9bebfa99c777e7d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:23.235078 11284 tablet_bootstrap.cc:492] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: Bootstrap starting.
I20260810 23:05:23.236709 11285 tablet_bootstrap.cc:492] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756: Bootstrap starting.
I20260810 23:05:23.237488 11284 tablet_bootstrap.cc:654] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:23.237738 11286 tablet_bootstrap.cc:492] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: Bootstrap starting.
I20260810 23:05:23.238900 11284 log.cc:824] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:23.239269 11285 tablet_bootstrap.cc:654] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:23.239589 11286 tablet_bootstrap.cc:654] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:23.240192 11285 log.cc:824] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:23.240899 11286 log.cc:824] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:23.241585 11284 tablet_bootstrap.cc:492] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: No bootstrap required, opened a new log
I20260810 23:05:23.241802 11284 ts_tablet_manager.cc:1397] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 23:05:23.242213 11285 tablet_bootstrap.cc:492] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756: No bootstrap required, opened a new log
I20260810 23:05:23.242403 11285 ts_tablet_manager.cc:1397] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:05:23.242995 11286 tablet_bootstrap.cc:492] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: No bootstrap required, opened a new log
I20260810 23:05:23.243186 11286 ts_tablet_manager.cc:1397] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:05:23.245959 11285 raft_consensus.cc:348] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } }
I20260810 23:05:23.245976 11284 raft_consensus.cc:348] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } }
I20260810 23:05:23.246229 11285 raft_consensus.cc:374] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:23.246284 11284 raft_consensus.cc:374] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:23.246300 11285 raft_consensus.cc:729] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d231d9bebdfd4f7bb14c08eb96782756, State: Initialized, Role: FOLLOWER
I20260810 23:05:23.246332 11284 raft_consensus.cc:729] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e29730fe814e838874a19ff2763393, State: Initialized, Role: FOLLOWER
I20260810 23:05:23.246840 11284 consensus_queue.cc:260] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } }
I20260810 23:05:23.246840 11285 consensus_queue.cc:260] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } }
I20260810 23:05:23.247052 11286 raft_consensus.cc:348] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } }
I20260810 23:05:23.247351 11286 raft_consensus.cc:374] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:23.247404 11286 raft_consensus.cc:729] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e5619de4fd47578ec03c1e7925795b, State: Initialized, Role: FOLLOWER
I20260810 23:05:23.247934 11286 consensus_queue.cc:260] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } }
I20260810 23:05:23.248095 11284 ts_tablet_manager.cc:1428] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:05:23.248158 11285 ts_tablet_manager.cc:1428] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:05:23.248507 11065 heartbeater.cc:503] Master 127.15.56.62:41031 was elected leader, sending a full tablet report...
I20260810 23:05:23.248652 10866 heartbeater.cc:503] Master 127.15.56.62:41031 was elected leader, sending a full tablet report...
I20260810 23:05:23.248927 11286 ts_tablet_manager.cc:1428] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:05:23.249275 11259 heartbeater.cc:503] Master 127.15.56.62:41031 was elected leader, sending a full tablet report...
W20260810 23:05:23.404882 11066 tablet.cc:2369] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:05:23.411834 11260 tablet.cc:2369] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:05:23.426057 10870 tablet.cc:2369] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:05:23.539255 11294 raft_consensus.cc:482] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:23.539535 11294 raft_consensus.cc:504] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } }
I20260810 23:05:23.539870 11293 raft_consensus.cc:482] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:23.540060 11293 raft_consensus.cc:504] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } }
I20260810 23:05:23.540933 11294 leader_election.cc:288] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e29730fe814e838874a19ff2763393 (127.15.56.2:36959), d231d9bebdfd4f7bb14c08eb96782756 (127.15.56.1:37171)
I20260810 23:05:23.542528 11293 leader_election.cc:288] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43e29730fe814e838874a19ff2763393 (127.15.56.2:36959), 80e5619de4fd47578ec03c1e7925795b (127.15.56.3:43041)
I20260810 23:05:23.546414 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.541328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:47245 (local address 127.15.56.2:36959)
0810 23:05:23.541461 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:23.541465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:23.541895 (+ 430us) server_negotiation.cc:407] Connection header received
0810 23:05:23.542090 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:23.542093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:23.542153 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:23.542208 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:23.542656 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.543451 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.545050 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.545670 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.545756 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.545836 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:23.546181 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 23:05:23.546267 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":63}
I20260810 23:05:23.546414 11278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.541669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:34489 (local address 127.15.56.1:37171)
0810 23:05:23.541775 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:23.541779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:23.541990 (+ 211us) server_negotiation.cc:407] Connection header received
0810 23:05:23.542225 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:23.542228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:23.542273 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:23.542324 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:23.542743 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.543440 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.545046 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.545713 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.545791 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.545840 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:23.546190 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 23:05:23.546266 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":32}
I20260810 23:05:23.546454 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.541225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:36959 (local address 127.15.56.3:44855)
0810 23:05:23.541433 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:23.541449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:23.541560 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:23.542193 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:23.542197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:23.542215 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:23.542349 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.542357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.543923 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.543926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.545058 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.545069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.545997 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.546001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.546123 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.546149 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:05:23.546196 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 23:05:23.546309 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":24}
I20260810 23:05:23.546792 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.541248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:37171 (local address 127.15.56.3:34489)
0810 23:05:23.541903 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:23.541918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:23.542070 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:23.542460 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:23.542463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:23.542479 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:23.542595 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.542615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.543535 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.543539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.544891 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.544903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.545920 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.545923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.546042 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.546072 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:05:23.546579 (+ 507us) client_negotiation.cc:240] Negotiation successful
0810 23:05:23.546661 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":142,"threads_started":1}
I20260810 23:05:23.546999 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.542390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:52121 (local address 127.15.56.3:43041)
0810 23:05:23.542492 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:23.542496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:23.542900 (+ 404us) server_negotiation.cc:407] Connection header received
0810 23:05:23.543064 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:23.543067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:23.543119 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:23.543163 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:23.543506 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.544240 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.545751 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.546348 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.546414 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.546465 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:23.546830 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 23:05:23.546900 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":45}
I20260810 23:05:23.547256 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.541314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:36959 (local address 127.15.56.1:47245)
0810 23:05:23.541808 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:23.541823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:23.541984 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:23.542355 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:23.542358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:23.542375 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:23.542541 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.542548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.543561 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.543564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.544891 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.544903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.545931 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.545934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.546028 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.546051 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:05:23.546711 (+ 660us) client_negotiation.cc:240] Negotiation successful
0810 23:05:23.546782 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260810 23:05:23.547293 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.541533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:44855 (local address 127.15.56.2:36959)
0810 23:05:23.541906 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:23.541909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:23.541919 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:05:23.541970 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:23.541974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:23.542043 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:23.542462 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:23.542549 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.543804 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.545223 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.545868 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.547016 (+ 1148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.547071 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:23.547117 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 23:05:23.547181 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":310,"thread_start_us":137,"threads_started":1}
I20260810 23:05:23.547415 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.542008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:43041 (local address 127.15.56.1:52121)
0810 23:05:23.542824 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:23.542838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:23.542962 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:23.543279 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:23.543282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:23.543296 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:23.543395 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.543400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.544369 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.544372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.545615 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:23.545626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.546564 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.546567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:23.546686 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.546711 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:05:23.547153 (+ 442us) client_negotiation.cc:240] Negotiation successful
0810 23:05:23.547247 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":35,"thread_start_us":125,"threads_started":1}
I20260810 23:05:23.547513 10799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d749a04bf1164bc9bebfa99c777e7d24" candidate_uuid: "80e5619de4fd47578ec03c1e7925795b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d231d9bebdfd4f7bb14c08eb96782756" is_pre_election: true
I20260810 23:05:23.547809 10799 raft_consensus.cc:2454] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80e5619de4fd47578ec03c1e7925795b in term 0.
I20260810 23:05:23.547868 10997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d749a04bf1164bc9bebfa99c777e7d24" candidate_uuid: "d231d9bebdfd4f7bb14c08eb96782756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e29730fe814e838874a19ff2763393" is_pre_election: true
I20260810 23:05:23.547866 10996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d749a04bf1164bc9bebfa99c777e7d24" candidate_uuid: "80e5619de4fd47578ec03c1e7925795b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e29730fe814e838874a19ff2763393" is_pre_election: true
I20260810 23:05:23.548027 11192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d749a04bf1164bc9bebfa99c777e7d24" candidate_uuid: "d231d9bebdfd4f7bb14c08eb96782756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e5619de4fd47578ec03c1e7925795b" is_pre_election: true
I20260810 23:05:23.548239 10997 raft_consensus.cc:2454] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d231d9bebdfd4f7bb14c08eb96782756 in term 0.
I20260810 23:05:23.548249 11192 raft_consensus.cc:2454] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d231d9bebdfd4f7bb14c08eb96782756 in term 0.
I20260810 23:05:23.548317 11094 leader_election.cc:302] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80e5619de4fd47578ec03c1e7925795b, d231d9bebdfd4f7bb14c08eb96782756; no voters:
I20260810 23:05:23.548615 11294 raft_consensus.cc:2781] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:05:23.548712 11294 raft_consensus.cc:482] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:05:23.548679 10705 leader_election.cc:302] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43e29730fe814e838874a19ff2763393, d231d9bebdfd4f7bb14c08eb96782756; no voters:
I20260810 23:05:23.548758 11294 raft_consensus.cc:3037] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:23.549139 11293 raft_consensus.cc:2781] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:05:23.549242 11293 raft_consensus.cc:482] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:05:23.549292 11293 raft_consensus.cc:3037] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:23.549966 11294 raft_consensus.cc:504] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } }
I20260810 23:05:23.550316 11294 leader_election.cc:288] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [CANDIDATE]: Term 1 election: Requested vote from peers 43e29730fe814e838874a19ff2763393 (127.15.56.2:36959), d231d9bebdfd4f7bb14c08eb96782756 (127.15.56.1:37171)
I20260810 23:05:23.550326 11293 raft_consensus.cc:504] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } }
I20260810 23:05:23.550681 10799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d749a04bf1164bc9bebfa99c777e7d24" candidate_uuid: "80e5619de4fd47578ec03c1e7925795b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d231d9bebdfd4f7bb14c08eb96782756"
I20260810 23:05:23.550740 11293 leader_election.cc:288] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [CANDIDATE]: Term 1 election: Requested vote from peers 43e29730fe814e838874a19ff2763393 (127.15.56.2:36959), 80e5619de4fd47578ec03c1e7925795b (127.15.56.3:43041)
I20260810 23:05:23.550732 10996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d749a04bf1164bc9bebfa99c777e7d24" candidate_uuid: "80e5619de4fd47578ec03c1e7925795b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e29730fe814e838874a19ff2763393"
I20260810 23:05:23.550861 10799 raft_consensus.cc:2379] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 80e5619de4fd47578ec03c1e7925795b in current term 1: Already voted for candidate d231d9bebdfd4f7bb14c08eb96782756 in this term.
I20260810 23:05:23.550866 10996 raft_consensus.cc:3037] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:23.551095 10997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d749a04bf1164bc9bebfa99c777e7d24" candidate_uuid: "d231d9bebdfd4f7bb14c08eb96782756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43e29730fe814e838874a19ff2763393"
I20260810 23:05:23.551182 11192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d749a04bf1164bc9bebfa99c777e7d24" candidate_uuid: "d231d9bebdfd4f7bb14c08eb96782756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e5619de4fd47578ec03c1e7925795b"
I20260810 23:05:23.551342 11192 raft_consensus.cc:2379] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d231d9bebdfd4f7bb14c08eb96782756 in current term 1: Already voted for candidate 80e5619de4fd47578ec03c1e7925795b in this term.
I20260810 23:05:23.551777 10705 leader_election.cc:302] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d231d9bebdfd4f7bb14c08eb96782756; no voters: 43e29730fe814e838874a19ff2763393, 80e5619de4fd47578ec03c1e7925795b
I20260810 23:05:23.551924 11293 raft_consensus.cc:2726] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:05:23.552201 10996 raft_consensus.cc:2454] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80e5619de4fd47578ec03c1e7925795b in term 1.
I20260810 23:05:23.552481 11095 leader_election.cc:302] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43e29730fe814e838874a19ff2763393, 80e5619de4fd47578ec03c1e7925795b; no voters: d231d9bebdfd4f7bb14c08eb96782756
I20260810 23:05:23.552681 11294 raft_consensus.cc:2781] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:23.552932 11294 raft_consensus.cc:686] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 1 LEADER]: Becoming Leader. State: Replica: 80e5619de4fd47578ec03c1e7925795b, State: Running, Role: LEADER
I20260810 23:05:23.553257 11294 consensus_queue.cc:237] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } }
I20260810 23:05:23.556818 10593 catalog_manager.cc:5495] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b reported cstate change: term changed from 0 to 1, leader changed from <none> to 80e5619de4fd47578ec03c1e7925795b (127.15.56.3). New cstate: current_term: 1 leader_uuid: "80e5619de4fd47578ec03c1e7925795b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80e5619de4fd47578ec03c1e7925795b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43041 } health_report { overall_health: HEALTHY } } }
I20260810 23:05:23.591346 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.587714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.15.56.3:43041)
0810 23:05:23.587860 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:23.587864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:23.587880 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:05:23.587970 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:23.587972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:23.588027 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:23.588121 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:05:23.588397 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.588911 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.589756 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.589975 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.590098 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.590425 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:05:23.590935 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:05:23.591050 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:23.591109 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 23:05:23.591169 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:05:23.597590 10799 raft_consensus.cc:1261] T d749a04bf1164bc9bebfa99c777e7d24 P d231d9bebdfd4f7bb14c08eb96782756 [term 1 FOLLOWER]: Refusing update from remote peer 80e5619de4fd47578ec03c1e7925795b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:23.597589 10996 raft_consensus.cc:1261] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393 [term 1 FOLLOWER]: Refusing update from remote peer 80e5619de4fd47578ec03c1e7925795b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:23.598412 11300 consensus_queue.cc:1035] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d231d9bebdfd4f7bb14c08eb96782756" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:23.598632 11294 consensus_queue.cc:1035] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43e29730fe814e838874a19ff2763393" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:23.607113 11311 mvcc.cc:205] Tried to move back new op lower bound from 7317107194248216576 to 7317107194076798976. Current Snapshot: MvccSnapshot[applied={T|T < 7317107194248216576}]
I20260810 23:05:23.607651 11310 mvcc.cc:205] Tried to move back new op lower bound from 7317107194248216576 to 7317107194076798976. Current Snapshot: MvccSnapshot[applied={T|T < 7317107194248216576}]
I20260810 23:05:23.616253 10592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:23.616444 10592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:23.619351 10592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c51d3153b91145ee827cb7f22a6f28b3: Sending DeleteTablet for 3 replicas of tablet d749a04bf1164bc9bebfa99c777e7d24
I20260810 23:05:23.620363 10771 tablet_service.cc:1514] Processing DeleteTablet for tablet d749a04bf1164bc9bebfa99c777e7d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:23 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:39152
I20260810 23:05:23.620381 10967 tablet_service.cc:1514] Processing DeleteTablet for tablet d749a04bf1164bc9bebfa99c777e7d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:23 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:39766
I20260810 23:05:23.620759 11166 tablet_service.cc:1514] Processing DeleteTablet for tablet d749a04bf1164bc9bebfa99c777e7d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:23 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:53688
I20260810 23:05:23.621157 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 10681
I20260810 23:05:23.624122 11321 tablet_replica.cc:330] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: stopping tablet replica
I20260810 23:05:23.624655 11321 raft_consensus.cc:2229] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:23.625151 11321 raft_consensus.cc:2258] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:23.636878 11323 tablet_replica.cc:330] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: stopping tablet replica
I20260810 23:05:23.637383 11323 raft_consensus.cc:2229] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:05:23.638087 11323 raft_consensus.cc:2258] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:23.639370 11321 ts_tablet_manager.cc:1905] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:23.639765 11323 ts_tablet_manager.cc:1905] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:23.642654 11323 ts_tablet_manager.cc:1918] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:23.642656 11321 ts_tablet_manager.cc:1918] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:23.642767 11323 log.cc:1196] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/d749a04bf1164bc9bebfa99c777e7d24
I20260810 23:05:23.642949 11321 log.cc:1196] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/d749a04bf1164bc9bebfa99c777e7d24
I20260810 23:05:23.643085 11323 ts_tablet_manager.cc:1939] T d749a04bf1164bc9bebfa99c777e7d24 P 80e5619de4fd47578ec03c1e7925795b: Deleting consensus metadata
I20260810 23:05:23.643385 11321 ts_tablet_manager.cc:1939] T d749a04bf1164bc9bebfa99c777e7d24 P 43e29730fe814e838874a19ff2763393: Deleting consensus metadata
I20260810 23:05:23.644582 10574 catalog_manager.cc:4841] TS 80e5619de4fd47578ec03c1e7925795b (127.15.56.3:43041): tablet d749a04bf1164bc9bebfa99c777e7d24 (table test-table [id=9aabf365c78d49c9ad273e31a63252dc]) successfully deleted
I20260810 23:05:23.644850 10574 catalog_manager.cc:4841] TS 43e29730fe814e838874a19ff2763393 (127.15.56.2:36959): tablet d749a04bf1164bc9bebfa99c777e7d24 (table test-table [id=9aabf365c78d49c9ad273e31a63252dc]) successfully deleted
W20260810 23:05:23.646802 10573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:37171 (error 108)
I20260810 23:05:23.646960 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 10877
I20260810 23:05:23.653620 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.650176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:37171 (local address 127.0.0.1:39160)
0810 23:05:23.653409 (+ 3233us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:23.653504 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3114}
W20260810 23:05:23.653939 10573 catalog_manager.cc:4584] TS d231d9bebdfd4f7bb14c08eb96782756 (127.15.56.1:37171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d749a04bf1164bc9bebfa99c777e7d24: Network error: Client connection negotiation failed: client connection to 127.15.56.1:37171: connect: Connection refused (error 111)
I20260810 23:05:23.662335 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 11073
I20260810 23:05:23.681622 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 10553
2026-08-10T23:05:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2845 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T23:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:23Z Disabled control of system clock
I20260810 23:05:23.726179 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:38827
--webserver_interface=127.15.56.62
--webserver_port=0
--builtin_ntp_servers=127.15.56.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:38827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:05:23.847294 11329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:23.847630 11329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:23.847707 11329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:23.851890 11329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:23.851961 11329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:23.851975 11329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:23.851991 11329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:23.852006 11329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:23.857085 11329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:38827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:38827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:23.858302 11329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:23.859824 11329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:23.868861 11338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:23.868875 11336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:23.868861 11335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:23.869043 11329 server_base.cc:1034] running on GCE node
I20260810 23:05:23.869657 11329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:23.870844 11329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:23.872041 11329 hybrid_clock.cc:648] HybridClock initialized: now 1786403123872005 us; error 48 us; skew 500 ppm
I20260810 23:05:23.874066 11329 webserver.cc:460] Webserver started at http://127.15.56.62:44283/ using document root <none> and password file <none>
I20260810 23:05:23.874687 11329 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:23.874764 11329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:23.874984 11329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:23.877388 11329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "78359e350b5a49a688871ea4c34be92c"
format_stamp: "Formatted at 2026-08-10 23:05:23 on dist-test-slave-6bbx"
server_key: "9cb1fc5050fb4c2ba69eb3d7e68b2723"
server_key_iv: "0ad9ddd3592d03c6c03ab0a480b9f803"
server_key_version: "encryptionkey@0"
I20260810 23:05:23.878019 11329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "78359e350b5a49a688871ea4c34be92c"
format_stamp: "Formatted at 2026-08-10 23:05:23 on dist-test-slave-6bbx"
server_key: "9cb1fc5050fb4c2ba69eb3d7e68b2723"
server_key_iv: "0ad9ddd3592d03c6c03ab0a480b9f803"
server_key_version: "encryptionkey@0"
I20260810 23:05:23.882319 11329 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:05:23.885262 11344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:23.886935 11329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:05:23.887130 11329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "78359e350b5a49a688871ea4c34be92c"
format_stamp: "Formatted at 2026-08-10 23:05:23 on dist-test-slave-6bbx"
server_key: "9cb1fc5050fb4c2ba69eb3d7e68b2723"
server_key_iv: "0ad9ddd3592d03c6c03ab0a480b9f803"
server_key_version: "encryptionkey@0"
I20260810 23:05:23.887274 11329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:23.911032 11329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:23.911960 11329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:23.912207 11329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:23.919776 11329 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:38827
I20260810 23:05:23.919799 11426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:38827 every 8 connection(s)
I20260810 23:05:23.920841 11329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:23.923646 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 11329
I20260810 23:05:23.923892 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:23.924283 15584 external_mini_cluster.cc:1428] Setting key b69bd67a7ad166018cb499fdcca10d09
I20260810 23:05:23.924796 11429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:23.933168 11429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c: Bootstrap starting.
I20260810 23:05:23.935720 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:23.925983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.15.56.62:38827)
0810 23:05:23.926722 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:23.926741 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:23.926786 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:05:23.928038 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:23.928066 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:23.928706 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:23.929081 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:23.929526 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.930564 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.931917 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:23.932504 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:23.932648 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:23.933704 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:23.933710 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:23.933726 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:23.934178 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:23.934243 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:23.934266 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:23.934375 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:23.934532 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 23:05:23.934835 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":150,"threads_started":1}
I20260810 23:05:23.935927 11429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:23.936905 11429 log.cc:824] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:23.939304 11429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c: No bootstrap required, opened a new log
I20260810 23:05:23.943300 11429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78359e350b5a49a688871ea4c34be92c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38827 } }
I20260810 23:05:23.943593 11429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:23.943650 11429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78359e350b5a49a688871ea4c34be92c, State: Initialized, Role: FOLLOWER
I20260810 23:05:23.944141 11429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78359e350b5a49a688871ea4c34be92c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38827 } }
I20260810 23:05:23.944310 11429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:23.944382 11429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:23.944494 11429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:23.945684 11429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78359e350b5a49a688871ea4c34be92c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38827 } }
I20260810 23:05:23.946075 11429 leader_election.cc:302] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78359e350b5a49a688871ea4c34be92c; no voters:
I20260810 23:05:23.946447 11429 leader_election.cc:288] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:23.946604 11438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:23.946867 11438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [term 1 LEADER]: Becoming Leader. State: Replica: 78359e350b5a49a688871ea4c34be92c, State: Running, Role: LEADER
I20260810 23:05:23.947178 11438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78359e350b5a49a688871ea4c34be92c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38827 } }
I20260810 23:05:23.947875 11429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:23.949119 11440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78359e350b5a49a688871ea4c34be92c. Latest consensus state: current_term: 1 leader_uuid: "78359e350b5a49a688871ea4c34be92c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78359e350b5a49a688871ea4c34be92c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38827 } } }
I20260810 23:05:23.949096 11439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78359e350b5a49a688871ea4c34be92c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78359e350b5a49a688871ea4c34be92c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 38827 } } }
I20260810 23:05:23.949306 11439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:23.949548 11440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:23.950045 11447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:23.953358 11447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:23.960702 11447 catalog_manager.cc:1347] Generated new cluster ID: 7ec4b63b216449e29b350c1557ab236a
I20260810 23:05:23.960798 11447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:23.989449 11447 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:23.990881 11447 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:24.007398 11447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c: Generated new TSK 0
I20260810 23:05:24.008443 11447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:24.011858 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:38827
--builtin_ntp_servers=127.15.56.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:05:24.134356 11460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:24.134697 11460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:24.134757 11460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:24.139019 11460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:24.139096 11460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:24.139178 11460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:05:24.144387 11460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:24.145660 11460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:24.147236 11460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:05:24.158550 11460 server_base.cc:1034] running on GCE node
W20260810 23:05:24.158514 11470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:24.158504 11473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:24.158505 11476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:24.159302 11460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:24.160171 11460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:24.161396 11460 hybrid_clock.cc:648] HybridClock initialized: now 1786403124161371 us; error 37 us; skew 500 ppm
I20260810 23:05:24.163501 11460 webserver.cc:460] Webserver started at http://127.15.56.1:37195/ using document root <none> and password file <none>
I20260810 23:05:24.164283 11460 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:24.164424 11460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:24.164685 11460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:24.166957 11460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "f60db0466ea247d3a0fe5a0b1e377dd1"
format_stamp: "Formatted at 2026-08-10 23:05:24 on dist-test-slave-6bbx"
server_key: "64e83234c2dc3a5949cf546175644b39"
server_key_iv: "f9e357c0a993bf7609ea3902135bfb0d"
server_key_version: "encryptionkey@0"
I20260810 23:05:24.167580 11460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "f60db0466ea247d3a0fe5a0b1e377dd1"
format_stamp: "Formatted at 2026-08-10 23:05:24 on dist-test-slave-6bbx"
server_key: "64e83234c2dc3a5949cf546175644b39"
server_key_iv: "f9e357c0a993bf7609ea3902135bfb0d"
server_key_version: "encryptionkey@0"
I20260810 23:05:24.172008 11460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:24.174875 11483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:24.176342 11460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:24.176460 11460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "f60db0466ea247d3a0fe5a0b1e377dd1"
format_stamp: "Formatted at 2026-08-10 23:05:24 on dist-test-slave-6bbx"
server_key: "64e83234c2dc3a5949cf546175644b39"
server_key_iv: "f9e357c0a993bf7609ea3902135bfb0d"
server_key_version: "encryptionkey@0"
I20260810 23:05:24.176564 11460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:24.195281 11460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:24.196040 11460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:24.196205 11460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:24.196769 11460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:24.197714 11460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:24.197760 11460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:24.197799 11460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:24.197819 11460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:24.206969 11460 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:43469
I20260810 23:05:24.207041 11639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:43469 every 8 connection(s)
I20260810 23:05:24.208257 11460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:24.211472 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 11460
I20260810 23:05:24.211613 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:05:24.211979 15584 external_mini_cluster.cc:1428] Setting key 4ec2181ee8f6107363e57e4b5f4e6113
I20260810 23:05:24.214082 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:38827
--builtin_ntp_servers=127.15.56.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:05:24.218984 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.210458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:53497 (local address 127.15.56.62:38827)
0810 23:05:24.210638 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:24.210643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:24.212075 (+ 1432us) server_negotiation.cc:407] Connection header received
0810 23:05:24.213754 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:24.213757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:24.213813 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:24.213892 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:24.214940 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.215469 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.216808 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.217126 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.217240 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.218440 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:24.218445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:24.218448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:24.218467 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:24.218496 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:24.218508 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:24.218595 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:24.218715 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:05:24.218811 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:05:24.220039 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.210912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:38827 (local address 127.15.56.1:53497)
0810 23:05:24.211896 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:24.211951 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:24.213518 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:24.214095 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:24.214103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:24.214222 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:24.214793 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.214807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.215629 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.215633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.216631 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.216642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.217307 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.217311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.217487 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.218140 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:05:24.218141 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:24.218188 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:05:24.218261 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:24.218591 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:24.218596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:24.218600 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:05:24.218697 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:05:24.219024 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":150,"threads_started":1}
I20260810 23:05:24.221392 11640 heartbeater.cc:347] Connected to a master server at 127.15.56.62:38827
I20260810 23:05:24.221650 11640 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:24.222240 11640 heartbeater.cc:511] Master 127.15.56.62:38827 requested a full tablet report, sending...
I20260810 23:05:24.223927 11374 ts_manager.cc:194] Registered new tserver with Master: f60db0466ea247d3a0fe5a0b1e377dd1 (127.15.56.1:43469)
I20260810 23:05:24.225131 11374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.1:53497
W20260810 23:05:24.334519 11644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:24.334882 11644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:24.334945 11644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:24.339248 11644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:24.339341 11644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:24.339419 11644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:05:24.344602 11644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:24.345894 11644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:24.347426 11644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:24.358332 11662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:24.358367 11658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:24.358547 11644 server_base.cc:1034] running on GCE node
W20260810 23:05:24.358355 11655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:24.359218 11644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:24.360013 11644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:24.361191 11644 hybrid_clock.cc:648] HybridClock initialized: now 1786403124361164 us; error 41 us; skew 500 ppm
I20260810 23:05:24.363407 11644 webserver.cc:460] Webserver started at http://127.15.56.2:37619/ using document root <none> and password file <none>
I20260810 23:05:24.364106 11644 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:24.364183 11644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:24.364390 11644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:24.366665 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "08f6e81b94ea412aaf7c328520badb3d"
format_stamp: "Formatted at 2026-08-10 23:05:24 on dist-test-slave-6bbx"
server_key: "03fd01cabcc26f56f3289f704c4bcb13"
server_key_iv: "bd1ebacc5ca5e01dce1120207823ad77"
server_key_version: "encryptionkey@0"
I20260810 23:05:24.367264 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "08f6e81b94ea412aaf7c328520badb3d"
format_stamp: "Formatted at 2026-08-10 23:05:24 on dist-test-slave-6bbx"
server_key: "03fd01cabcc26f56f3289f704c4bcb13"
server_key_iv: "bd1ebacc5ca5e01dce1120207823ad77"
server_key_version: "encryptionkey@0"
I20260810 23:05:24.371373 11644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:05:24.374387 11671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:24.375936 11644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:24.376062 11644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "08f6e81b94ea412aaf7c328520badb3d"
format_stamp: "Formatted at 2026-08-10 23:05:24 on dist-test-slave-6bbx"
server_key: "03fd01cabcc26f56f3289f704c4bcb13"
server_key_iv: "bd1ebacc5ca5e01dce1120207823ad77"
server_key_version: "encryptionkey@0"
I20260810 23:05:24.376202 11644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:24.398844 11644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:24.399645 11644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:24.399928 11644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:24.400601 11644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:24.401618 11644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:24.401669 11644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:24.401714 11644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:24.401856 11644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:24.411119 11644 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:45683
I20260810 23:05:24.411142 11838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:45683 every 8 connection(s)
I20260810 23:05:24.412209 11644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:24.412539 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 11644
I20260810 23:05:24.412652 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:05:24.413017 15584 external_mini_cluster.cc:1428] Setting key 29d72be096e8457cd902b55a6661e139
I20260810 23:05:24.415225 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:38827
--builtin_ntp_servers=127.15.56.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:05:24.422806 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.414228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:41457 (local address 127.15.56.62:38827)
0810 23:05:24.414390 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:24.414395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:24.415790 (+ 1395us) server_negotiation.cc:407] Connection header received
0810 23:05:24.417410 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:24.417414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:24.417469 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:24.417558 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:24.418504 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.419249 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.420544 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.420894 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.421010 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.422272 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:24.422277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:24.422280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:24.422297 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:24.422333 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:24.422344 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:24.422418 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:24.422559 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 23:05:24.422635 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:05:24.423787 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.414773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:38827 (local address 127.15.56.2:41457)
0810 23:05:24.415597 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:24.415702 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:24.417195 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:24.417744 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:24.417752 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:24.417845 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:24.418365 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.418379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.419409 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.419412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.420415 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.420427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.421084 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.421087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.421261 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.421966 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:05:24.421967 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:24.422013 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:05:24.422090 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:24.422430 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:24.422436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:24.422441 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:05:24.422540 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 23:05:24.422811 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":152,"threads_started":1}
I20260810 23:05:24.425123 11840 heartbeater.cc:347] Connected to a master server at 127.15.56.62:38827
I20260810 23:05:24.425380 11840 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:24.426000 11840 heartbeater.cc:511] Master 127.15.56.62:38827 requested a full tablet report, sending...
I20260810 23:05:24.427179 11374 ts_manager.cc:194] Registered new tserver with Master: 08f6e81b94ea412aaf7c328520badb3d (127.15.56.2:45683)
I20260810 23:05:24.427830 11374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.2:41457
W20260810 23:05:24.535300 11845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:24.535611 11845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:24.535652 11845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:24.539826 11845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:24.539901 11845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:24.539958 11845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:05:24.544970 11845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:24.546159 11845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:24.547657 11845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:24.558173 11857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:24.558274 11845 server_base.cc:1034] running on GCE node
W20260810 23:05:24.558175 11853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:24.558177 11854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:24.558972 11845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:24.559659 11845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:24.560891 11845 hybrid_clock.cc:648] HybridClock initialized: now 1786403124560855 us; error 43 us; skew 500 ppm
I20260810 23:05:24.562927 11845 webserver.cc:460] Webserver started at http://127.15.56.3:34041/ using document root <none> and password file <none>
I20260810 23:05:24.563516 11845 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:24.563607 11845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:24.563874 11845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:24.566061 11845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "5615292ecf5e47d2af45169a54f3c0fd"
format_stamp: "Formatted at 2026-08-10 23:05:24 on dist-test-slave-6bbx"
server_key: "8c2e6db6f984abfd2ba47aebfc5522c6"
server_key_iv: "d6c368009050dece26226eab2746ce30"
server_key_version: "encryptionkey@0"
I20260810 23:05:24.566663 11845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "5615292ecf5e47d2af45169a54f3c0fd"
format_stamp: "Formatted at 2026-08-10 23:05:24 on dist-test-slave-6bbx"
server_key: "8c2e6db6f984abfd2ba47aebfc5522c6"
server_key_iv: "d6c368009050dece26226eab2746ce30"
server_key_version: "encryptionkey@0"
I20260810 23:05:24.570847 11845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 23:05:24.573582 11866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:24.574970 11845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:05:24.575104 11845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "5615292ecf5e47d2af45169a54f3c0fd"
format_stamp: "Formatted at 2026-08-10 23:05:24 on dist-test-slave-6bbx"
server_key: "8c2e6db6f984abfd2ba47aebfc5522c6"
server_key_iv: "d6c368009050dece26226eab2746ce30"
server_key_version: "encryptionkey@0"
I20260810 23:05:24.575217 11845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:24.589675 11845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:24.590415 11845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:24.590593 11845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:24.591207 11845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:24.592308 11845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:24.592361 11845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:24.592417 11845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:24.592443 11845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:24.601086 11845 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:43541
I20260810 23:05:24.601176 12037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:43541 every 8 connection(s)
I20260810 23:05:24.602092 11845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:05:24.603170 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 11845
I20260810 23:05:24.603266 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:05:24.603588 15584 external_mini_cluster.cc:1428] Setting key a604479cd3ae81d7018e50c1d67f08ec
I20260810 23:05:24.612860 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.604445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:43083 (local address 127.15.56.62:38827)
0810 23:05:24.604594 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:24.604598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:24.606055 (+ 1457us) server_negotiation.cc:407] Connection header received
0810 23:05:24.607544 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:24.607548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:24.607601 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:24.607715 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:24.608765 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.609335 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.610596 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.610851 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.610951 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.612329 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:24.612332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:24.612334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:24.612349 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:24.612378 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:24.612387 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:24.612456 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:24.612660 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 23:05:24.612727 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:05:24.613801 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.605043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:38827 (local address 127.15.56.3:43083)
0810 23:05:24.605884 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:24.605949 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:24.607315 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:24.607900 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:24.607908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:24.608002 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:24.608606 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.608622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.609473 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.609476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.610461 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.610472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.611086 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.611091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.611275 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.612008 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:05:24.612010 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:24.612057 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:05:24.612137 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:24.612465 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:24.612473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:24.612478 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:05:24.612581 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:05:24.612857 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":153,"threads_started":1}
I20260810 23:05:24.615142 12039 heartbeater.cc:347] Connected to a master server at 127.15.56.62:38827
I20260810 23:05:24.615439 12039 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:24.616114 12039 heartbeater.cc:511] Master 127.15.56.62:38827 requested a full tablet report, sending...
I20260810 23:05:24.617434 11374 ts_manager.cc:194] Registered new tserver with Master: 5615292ecf5e47d2af45169a54f3c0fd (127.15.56.3:43541)
I20260810 23:05:24.618194 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:05:24.618172 11374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.3:43083
I20260810 23:05:24.624662 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.620587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.15.56.62:38827)
0810 23:05:24.620715 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:24.620719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:24.620895 (+ 176us) server_negotiation.cc:407] Connection header received
0810 23:05:24.620965 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:24.620967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:24.621013 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:24.621075 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:24.621352 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.621912 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.622907 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.623140 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.623237 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.624123 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:24.624126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:24.624128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:24.624144 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:24.624167 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:24.624174 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:24.624249 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:24.624468 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 23:05:24.624534 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:05:24.628600 11374 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:05:24.631958 11374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:05:24.647017 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.641035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:45683 (local address 127.0.0.1:36788)
0810 23:05:24.641454 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:24.641507 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:24.641639 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:24.642506 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:24.642514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:24.642536 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:24.642718 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.642726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.644133 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.644136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.645314 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.645326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.646390 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.646394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.646538 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.646589 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 23:05:24.646697 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 23:05:24.646848 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
I20260810 23:05:24.647197 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.641986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.15.56.3:43541)
0810 23:05:24.642791 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:24.642803 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:24.642838 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:05:24.642915 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:24.642921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:24.643075 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:24.643207 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:24.643445 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.644546 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.645917 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.646593 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.646707 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.646896 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:24.646949 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 23:05:24.647044 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":171,"threads_started":1}
I20260810 23:05:24.647514 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.641227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.15.56.2:45683)
0810 23:05:24.641924 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:24.641942 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:24.641971 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:05:24.642059 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:24.642065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:24.642321 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:24.642894 (+ 573us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:24.643039 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.644036 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.645460 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.646271 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.646378 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.646584 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:24.647243 (+ 659us) server_negotiation.cc:299] Negotiation successful
0810 23:05:24.647362 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":168,"threads_started":1}
I20260810 23:05:24.647696 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.641249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:43541 (local address 127.0.0.1:60934)
0810 23:05:24.642455 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:24.642469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:24.642561 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:24.643205 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:24.643208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:24.643224 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:24.643330 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.643338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.644674 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.644677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.645791 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.645801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.646723 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.646726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.646818 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.646842 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:05:24.646906 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 23:05:24.646976 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":116,"threads_started":1}
I20260810 23:05:24.648249 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.641899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:43469 (local address 127.0.0.1:54406)
0810 23:05:24.642728 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:24.642742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:24.642826 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:24.643495 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:24.643498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:24.643515 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:24.643604 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.643610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.644896 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.644899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.645922 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.645932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.647086 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.647090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.647942 (+ 852us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.647968 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:05:24.648039 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 23:05:24.648110 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260810 23:05:24.649040 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.642064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.15.56.1:43469)
0810 23:05:24.642480 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:24.642493 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:24.642919 (+ 426us) server_negotiation.cc:407] Connection header received
0810 23:05:24.643006 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:24.643011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:24.643211 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:24.643357 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:24.643741 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.644793 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.646059 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.646839 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.646940 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.647115 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:24.648092 (+ 977us) server_negotiation.cc:299] Negotiation successful
0810 23:05:24.648199 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":110,"threads_started":1}
I20260810 23:05:24.650851 11742 tablet_service.cc:1467] Processing CreateTablet for tablet f00b0b069684435e97a99f91a2a5e341 (DEFAULT_TABLE table=test-table [id=d11b0743f683444e901ffa1ee4de34a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:24.650890 11940 tablet_service.cc:1467] Processing CreateTablet for tablet f00b0b069684435e97a99f91a2a5e341 (DEFAULT_TABLE table=test-table [id=d11b0743f683444e901ffa1ee4de34a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:24.652026 11742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00b0b069684435e97a99f91a2a5e341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:24.652031 11940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00b0b069684435e97a99f91a2a5e341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:24.652253 11543 tablet_service.cc:1467] Processing CreateTablet for tablet f00b0b069684435e97a99f91a2a5e341 (DEFAULT_TABLE table=test-table [id=d11b0743f683444e901ffa1ee4de34a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:24.653399 11543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00b0b069684435e97a99f91a2a5e341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:24.658805 12062 tablet_bootstrap.cc:492] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d: Bootstrap starting.
I20260810 23:05:24.660485 12063 tablet_bootstrap.cc:492] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: Bootstrap starting.
I20260810 23:05:24.661110 12066 tablet_bootstrap.cc:492] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1: Bootstrap starting.
I20260810 23:05:24.662737 12063 tablet_bootstrap.cc:654] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:24.662750 12062 tablet_bootstrap.cc:654] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:24.663326 12066 tablet_bootstrap.cc:654] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:24.663587 12063 log.cc:824] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:24.663614 12062 log.cc:824] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:24.664142 12066 log.cc:824] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:24.665793 12062 tablet_bootstrap.cc:492] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d: No bootstrap required, opened a new log
I20260810 23:05:24.665793 12063 tablet_bootstrap.cc:492] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: No bootstrap required, opened a new log
I20260810 23:05:24.665838 12066 tablet_bootstrap.cc:492] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1: No bootstrap required, opened a new log
I20260810 23:05:24.666009 12066 ts_tablet_manager.cc:1397] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:05:24.666007 12062 ts_tablet_manager.cc:1397] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 23:05:24.666007 12063 ts_tablet_manager.cc:1397] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 23:05:24.669719 12066 raft_consensus.cc:348] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f6e81b94ea412aaf7c328520badb3d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 45683 } } peers { permanent_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 43469 } } peers { permanent_uuid: "5615292ecf5e47d2af45169a54f3c0fd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43541 } }
I20260810 23:05:24.669986 12066 raft_consensus.cc:374] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:24.670042 12066 raft_consensus.cc:729] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f60db0466ea247d3a0fe5a0b1e377dd1, State: Initialized, Role: FOLLOWER
I20260810 23:05:24.670503 12066 consensus_queue.cc:260] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f6e81b94ea412aaf7c328520badb3d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 45683 } } peers { permanent_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 43469 } } peers { permanent_uuid: "5615292ecf5e47d2af45169a54f3c0fd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43541 } }
I20260810 23:05:24.670409 12062 raft_consensus.cc:348] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f6e81b94ea412aaf7c328520badb3d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 45683 } } peers { permanent_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 43469 } } peers { permanent_uuid: "5615292ecf5e47d2af45169a54f3c0fd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43541 } }
I20260810 23:05:24.670409 12063 raft_consensus.cc:348] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f6e81b94ea412aaf7c328520badb3d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 45683 } } peers { permanent_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 43469 } } peers { permanent_uuid: "5615292ecf5e47d2af45169a54f3c0fd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43541 } }
I20260810 23:05:24.670678 12063 raft_consensus.cc:374] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:24.670678 12062 raft_consensus.cc:374] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:24.670738 12062 raft_consensus.cc:729] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f6e81b94ea412aaf7c328520badb3d, State: Initialized, Role: FOLLOWER
I20260810 23:05:24.670750 12063 raft_consensus.cc:729] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5615292ecf5e47d2af45169a54f3c0fd, State: Initialized, Role: FOLLOWER
I20260810 23:05:24.671278 11640 heartbeater.cc:503] Master 127.15.56.62:38827 was elected leader, sending a full tablet report...
I20260810 23:05:24.671231 12063 consensus_queue.cc:260] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f6e81b94ea412aaf7c328520badb3d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 45683 } } peers { permanent_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 43469 } } peers { permanent_uuid: "5615292ecf5e47d2af45169a54f3c0fd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43541 } }
I20260810 23:05:24.671231 12062 consensus_queue.cc:260] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f6e81b94ea412aaf7c328520badb3d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 45683 } } peers { permanent_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 43469 } } peers { permanent_uuid: "5615292ecf5e47d2af45169a54f3c0fd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43541 } }
I20260810 23:05:24.671622 12066 ts_tablet_manager.cc:1428] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:05:24.672272 12063 ts_tablet_manager.cc:1428] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260810 23:05:24.672272 12062 ts_tablet_manager.cc:1428] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 23:05:24.672621 11840 heartbeater.cc:503] Master 127.15.56.62:38827 was elected leader, sending a full tablet report...
I20260810 23:05:24.672647 12039 heartbeater.cc:503] Master 127.15.56.62:38827 was elected leader, sending a full tablet report...
W20260810 23:05:24.709692 11641 tablet.cc:2369] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:05:24.853133 12040 tablet.cc:2369] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:05:24.854061 12077 raft_consensus.cc:482] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:24.854281 12077 raft_consensus.cc:504] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f6e81b94ea412aaf7c328520badb3d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 45683 } } peers { permanent_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 43469 } } peers { permanent_uuid: "5615292ecf5e47d2af45169a54f3c0fd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43541 } }
I20260810 23:05:24.855515 12077 leader_election.cc:288] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f60db0466ea247d3a0fe5a0b1e377dd1 (127.15.56.1:43469), 5615292ecf5e47d2af45169a54f3c0fd (127.15.56.3:43541)
I20260810 23:05:24.860446 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.855814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:43469 (local address 127.15.56.2:60581)
0810 23:05:24.855965 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:24.855981 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:24.856088 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:24.856397 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:24.856400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:24.856415 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:24.856544 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.856550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.857575 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.857578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.858937 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.858950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.859908 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.859912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.860071 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.860119 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 23:05:24.860240 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:05:24.860300 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":47}
I20260810 23:05:24.860474 12060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.855910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:60581 (local address 127.15.56.1:43469)
0810 23:05:24.856047 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:24.856051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:24.856069 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:05:24.856232 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:24.856236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:24.856291 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:24.856371 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:24.856664 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.857424 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.859090 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.859724 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.859867 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.859934 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:24.860268 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 23:05:24.860344 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:05:24.860474 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.856064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:40247 (local address 127.15.56.3:43541)
0810 23:05:24.856206 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:24.856210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:24.856368 (+ 158us) server_negotiation.cc:407] Connection header received
0810 23:05:24.856478 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:24.856481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:24.856522 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:24.856579 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:24.856839 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.857564 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.859090 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.859767 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.859867 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.859934 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:24.860233 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 23:05:24.860345 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:05:24.860455 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.855835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:43541 (local address 127.15.56.2:40247)
0810 23:05:24.856278 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:24.856289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:24.856377 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:24.856616 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:24.856619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:24.856637 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:24.856740 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.856745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.857679 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.857689 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.858937 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:24.858950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.859896 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.859900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:24.860090 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.860126 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:05:24.860195 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 23:05:24.860279 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":124,"thread_start_us":137,"threads_started":1}
I20260810 23:05:24.861306 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00b0b069684435e97a99f91a2a5e341" candidate_uuid: "08f6e81b94ea412aaf7c328520badb3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5615292ecf5e47d2af45169a54f3c0fd" is_pre_election: true
I20260810 23:05:24.861272 11575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00b0b069684435e97a99f91a2a5e341" candidate_uuid: "08f6e81b94ea412aaf7c328520badb3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" is_pre_election: true
I20260810 23:05:24.861647 11575 raft_consensus.cc:2454] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08f6e81b94ea412aaf7c328520badb3d in term 0.
I20260810 23:05:24.861647 11962 raft_consensus.cc:2454] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08f6e81b94ea412aaf7c328520badb3d in term 0.
I20260810 23:05:24.862222 11676 leader_election.cc:302] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f6e81b94ea412aaf7c328520badb3d, 5615292ecf5e47d2af45169a54f3c0fd; no voters:
I20260810 23:05:24.862522 12077 raft_consensus.cc:2781] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:05:24.862615 12077 raft_consensus.cc:482] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:05:24.862676 12077 raft_consensus.cc:3037] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:24.863828 12077 raft_consensus.cc:504] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f6e81b94ea412aaf7c328520badb3d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 45683 } } peers { permanent_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 43469 } } peers { permanent_uuid: "5615292ecf5e47d2af45169a54f3c0fd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43541 } }
I20260810 23:05:24.864218 12077 leader_election.cc:288] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [CANDIDATE]: Term 1 election: Requested vote from peers f60db0466ea247d3a0fe5a0b1e377dd1 (127.15.56.1:43469), 5615292ecf5e47d2af45169a54f3c0fd (127.15.56.3:43541)
I20260810 23:05:24.864616 11575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00b0b069684435e97a99f91a2a5e341" candidate_uuid: "08f6e81b94ea412aaf7c328520badb3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1"
I20260810 23:05:24.864670 11962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00b0b069684435e97a99f91a2a5e341" candidate_uuid: "08f6e81b94ea412aaf7c328520badb3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5615292ecf5e47d2af45169a54f3c0fd"
I20260810 23:05:24.864746 11575 raft_consensus.cc:3037] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:24.864820 11962 raft_consensus.cc:3037] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:24.865856 11575 raft_consensus.cc:2454] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08f6e81b94ea412aaf7c328520badb3d in term 1.
I20260810 23:05:24.865921 11962 raft_consensus.cc:2454] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08f6e81b94ea412aaf7c328520badb3d in term 1.
I20260810 23:05:24.866225 11675 leader_election.cc:302] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f6e81b94ea412aaf7c328520badb3d, f60db0466ea247d3a0fe5a0b1e377dd1; no voters:
I20260810 23:05:24.866426 12077 raft_consensus.cc:2781] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:24.866680 12077 raft_consensus.cc:686] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 1 LEADER]: Becoming Leader. State: Replica: 08f6e81b94ea412aaf7c328520badb3d, State: Running, Role: LEADER
I20260810 23:05:24.867079 12077 consensus_queue.cc:237] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f6e81b94ea412aaf7c328520badb3d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 45683 } } peers { permanent_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 43469 } } peers { permanent_uuid: "5615292ecf5e47d2af45169a54f3c0fd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43541 } }
I20260810 23:05:24.870671 11372 catalog_manager.cc:5495] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 08f6e81b94ea412aaf7c328520badb3d (127.15.56.2). New cstate: current_term: 1 leader_uuid: "08f6e81b94ea412aaf7c328520badb3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f6e81b94ea412aaf7c328520badb3d" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 45683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 43469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5615292ecf5e47d2af45169a54f3c0fd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43541 } health_report { overall_health: UNKNOWN } } }
I20260810 23:05:24.892028 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.888310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.15.56.2:45683)
0810 23:05:24.888453 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:24.888457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:24.888491 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:05:24.888573 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:24.888575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:24.888637 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:24.888720 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:05:24.889005 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.889540 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.890419 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:24.890659 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:24.890751 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:24.891053 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:05:24.891537 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:05:24.891702 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:24.891786 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 23:05:24.891850 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:05:24.898600 11575 raft_consensus.cc:1261] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1 [term 1 FOLLOWER]: Refusing update from remote peer 08f6e81b94ea412aaf7c328520badb3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:24.898600 11962 raft_consensus.cc:1261] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd [term 1 FOLLOWER]: Refusing update from remote peer 08f6e81b94ea412aaf7c328520badb3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:24.899559 12079 consensus_queue.cc:1035] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5615292ecf5e47d2af45169a54f3c0fd" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:24.899822 12077 consensus_queue.cc:1035] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f60db0466ea247d3a0fe5a0b1e377dd1" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 43469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:24.909356 12084 mvcc.cc:205] Tried to move back new op lower bound from 7317107199575851008 to 7317107199458717696. Current Snapshot: MvccSnapshot[applied={T|T < 7317107199575851008}]
I20260810 23:05:24.918422 11374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:24.918632 11374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:24.921515 11374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78359e350b5a49a688871ea4c34be92c: Sending DeleteTablet for 3 replicas of tablet f00b0b069684435e97a99f91a2a5e341
I20260810 23:05:24.922405 11742 tablet_service.cc:1514] Processing DeleteTablet for tablet f00b0b069684435e97a99f91a2a5e341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:24 UTC) from {username='slave'} at 127.0.0.1:36788
I20260810 23:05:24.922613 11940 tablet_service.cc:1514] Processing DeleteTablet for tablet f00b0b069684435e97a99f91a2a5e341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:24 UTC) from {username='slave'} at 127.0.0.1:60934
I20260810 23:05:24.923069 12097 tablet_replica.cc:330] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d: stopping tablet replica
I20260810 23:05:24.923326 11543 tablet_service.cc:1514] Processing DeleteTablet for tablet f00b0b069684435e97a99f91a2a5e341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:24 UTC) from {username='slave'} at 127.0.0.1:54406
I20260810 23:05:24.923506 12097 raft_consensus.cc:2229] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:05:24.923978 12097 raft_consensus.cc:2258] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:24.924011 12099 tablet_replica.cc:330] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1: stopping tablet replica
I20260810 23:05:24.924368 12099 raft_consensus.cc:2229] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:24.924430 12098 tablet_replica.cc:330] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: stopping tablet replica
I20260810 23:05:24.924715 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 11460
I20260810 23:05:24.924755 12099 raft_consensus.cc:2258] T f00b0b069684435e97a99f91a2a5e341 P f60db0466ea247d3a0fe5a0b1e377dd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:24.928174 12097 ts_tablet_manager.cc:1905] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:24.929430 12098 raft_consensus.cc:2229] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:24.929929 12098 raft_consensus.cc:2258] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:24.931329 12097 ts_tablet_manager.cc:1918] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:24.931437 12097 log.cc:1196] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/f00b0b069684435e97a99f91a2a5e341
I20260810 23:05:24.931797 12097 ts_tablet_manager.cc:1939] T f00b0b069684435e97a99f91a2a5e341 P 08f6e81b94ea412aaf7c328520badb3d: Deleting consensus metadata
I20260810 23:05:24.934579 12098 ts_tablet_manager.cc:1905] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:24.937541 12098 ts_tablet_manager.cc:1918] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:24.937757 12098 log.cc:1196] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/f00b0b069684435e97a99f91a2a5e341
I20260810 23:05:24.938238 12098 ts_tablet_manager.cc:1939] T f00b0b069684435e97a99f91a2a5e341 P 5615292ecf5e47d2af45169a54f3c0fd: Deleting consensus metadata
I20260810 23:05:24.940106 11350 catalog_manager.cc:4841] TS 5615292ecf5e47d2af45169a54f3c0fd (127.15.56.3:43541): tablet f00b0b069684435e97a99f91a2a5e341 (table test-table [id=d11b0743f683444e901ffa1ee4de34a2]) successfully deleted
W20260810 23:05:24.952126 11348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:43469 (error 108)
I20260810 23:05:24.952456 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 11644
I20260810 23:05:24.953536 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.953111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:43469 (local address 127.0.0.1:54410)
0810 23:05:24.953340 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:24.953431 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:43469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260810 23:05:24.953739 11348 catalog_manager.cc:4584] TS f60db0466ea247d3a0fe5a0b1e377dd1 (127.15.56.1:43469): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f00b0b069684435e97a99f91a2a5e341: Network error: Client connection negotiation failed: client connection to 127.15.56.1:43469: connect: Connection refused (error 111)
I20260810 23:05:24.962179 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 11845
I20260810 23:05:24.965406 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:24.965019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:45683 (local address 127.0.0.1:36804)
0810 23:05:24.965256 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:24.965308 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.2:45683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260810 23:05:24.981009 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 11329
2026-08-10T23:05:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1294 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:25 dist-test-slave-6bbx krb5kdc[12103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:25 dist-test-slave-6bbx krb5kdc[12103](info): set up 2 sockets
Aug 10 23:05:25 dist-test-slave-6bbx krb5kdc[12103](info): commencing operation
krb5kdc: starting...
W20260810 23:05:26.162389 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:26 dist-test-slave-6bbx krb5kdc[12103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:05:26.298621 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43085
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:43085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:26.419595 12121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:26.419935 12121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:26.419993 12121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:26.424081 12121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:26.424149 12121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:26.424182 12121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:26.424216 12121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:26.424243 12121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:26.429710 12121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:43085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:26.430833 12121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:26.432008 12121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:26.441390 12130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:26.441391 12129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:26.441391 12133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:26.441977 12121 server_base.cc:1034] running on GCE node
I20260810 23:05:26.442633 12121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:26.443979 12121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:26.445204 12121 hybrid_clock.cc:648] HybridClock initialized: now 1786403126445188 us; error 53 us; skew 500 ppm
Aug 10 23:05:26 dist-test-slave-6bbx krb5kdc[12103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403126, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:26.448635 12121 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:05:26.449751 12121 webserver.cc:460] Webserver started at http://127.15.56.62:36129/ using document root <none> and password file <none>
I20260810 23:05:26.450367 12121 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:26.450444 12121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:26.450676 12121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:26.452980 12121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "37c744d70e4c4d0bb558149e8e062c3a"
format_stamp: "Formatted at 2026-08-10 23:05:26 on dist-test-slave-6bbx"
server_key: "3a4b3d02895982e853d182428480698f"
server_key_iv: "d297c894aaa47efe023aaa626a414071"
server_key_version: "encryptionkey@0"
I20260810 23:05:26.453579 12121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "37c744d70e4c4d0bb558149e8e062c3a"
format_stamp: "Formatted at 2026-08-10 23:05:26 on dist-test-slave-6bbx"
server_key: "3a4b3d02895982e853d182428480698f"
server_key_iv: "d297c894aaa47efe023aaa626a414071"
server_key_version: "encryptionkey@0"
I20260810 23:05:26.457828 12121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:05:26.460515 12145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:26.461895 12121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:05:26.462023 12121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "37c744d70e4c4d0bb558149e8e062c3a"
format_stamp: "Formatted at 2026-08-10 23:05:26 on dist-test-slave-6bbx"
server_key: "3a4b3d02895982e853d182428480698f"
server_key_iv: "d297c894aaa47efe023aaa626a414071"
server_key_version: "encryptionkey@0"
I20260810 23:05:26.462147 12121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:26.479179 12121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:26.482265 12121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:26.482455 12121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:26.489804 12121 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:43085
I20260810 23:05:26.489823 12223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:43085 every 8 connection(s)
I20260810 23:05:26.490855 12121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:26.493786 12224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:26.496886 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 12121
I20260810 23:05:26.497036 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:26.497313 15584 external_mini_cluster.cc:1428] Setting key 10611728a373a8c279fba868aeaa43a5
I20260810 23:05:26.500123 12224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a: Bootstrap starting.
I20260810 23:05:26.503050 12224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:26.503950 12224 log.cc:824] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:26.506242 12224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a: No bootstrap required, opened a new log
Aug 10 23:05:26 dist-test-slave-6bbx krb5kdc[12103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:05:26.509931 12224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c744d70e4c4d0bb558149e8e062c3a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43085 } }
I20260810 23:05:26.510169 12224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:26.510205 12224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37c744d70e4c4d0bb558149e8e062c3a, State: Initialized, Role: FOLLOWER
I20260810 23:05:26.510676 12224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c744d70e4c4d0bb558149e8e062c3a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43085 } }
I20260810 23:05:26.510819 12224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:26.510888 12224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:26.510988 12224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:26.512087 12224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c744d70e4c4d0bb558149e8e062c3a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43085 } }
I20260810 23:05:26.512475 12224 leader_election.cc:302] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37c744d70e4c4d0bb558149e8e062c3a; no voters:
I20260810 23:05:26.512812 12224 leader_election.cc:288] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:26.512920 12231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:26.513159 12231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [term 1 LEADER]: Becoming Leader. State: Replica: 37c744d70e4c4d0bb558149e8e062c3a, State: Running, Role: LEADER
I20260810 23:05:26.513477 12231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c744d70e4c4d0bb558149e8e062c3a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43085 } }
I20260810 23:05:26.513684 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:26.498796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.15.56.62:43085)
0810 23:05:26.499290 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:26.499310 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:26.499356 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:05:26.500215 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:26.500242 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:26.500699 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:26.501088 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:26.501846 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:26.503376 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:26.504456 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:26.505090 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:26.505295 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:26.508087 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:26.508093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:26.508108 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:26.508136 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:26.510389 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:26.510831 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:26.510837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:26.510845 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:26.511000 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:26.511319 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:26.511321 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:26.511322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:26.511777 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:26.511943 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:26.512278 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:05:26.512602 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":130,"threads_started":1}
I20260810 23:05:26.514112 12224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:26.514699 12231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37c744d70e4c4d0bb558149e8e062c3a. Latest consensus state: current_term: 1 leader_uuid: "37c744d70e4c4d0bb558149e8e062c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c744d70e4c4d0bb558149e8e062c3a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43085 } } }
I20260810 23:05:26.514827 12231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:26.515172 12233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37c744d70e4c4d0bb558149e8e062c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c744d70e4c4d0bb558149e8e062c3a" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43085 } } }
I20260810 23:05:26.515261 12233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a [sys.catalog]: This master's current role is: LEADER
W20260810 23:05:26.518610 12254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:05:26.518676 12254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:26.518749 12243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:26.519438 12243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:26.524653 12243 catalog_manager.cc:1347] Generated new cluster ID: 547fd4454a9642f699b4d189c8771c3a
I20260810 23:05:26.524710 12243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:26.537194 12243 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:26.538136 12243 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:26.543697 12243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a: Generated new TSK 0
I20260810 23:05:26.544452 12243 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:05:26.588325 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43085
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:26.711261 12265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:26.711560 12265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:26.711606 12265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:26.715912 12265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:26.715986 12265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:26.716051 12265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:05:26.721714 12265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:26.722957 12265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:26.724112 12265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:26.734755 12274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:26.734751 12273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:26.734953 12265 server_base.cc:1034] running on GCE node
W20260810 23:05:26.734766 12276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:26.735584 12265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:26.736513 12265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:26.737710 12265 hybrid_clock.cc:648] HybridClock initialized: now 1786403126737687 us; error 36 us; skew 500 ppm
Aug 10 23:05:26 dist-test-slave-6bbx krb5kdc[12103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403126, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:26.740891 12265 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:05:26.741930 12265 webserver.cc:460] Webserver started at http://127.15.56.1:46163/ using document root <none> and password file <none>
I20260810 23:05:26.742553 12265 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:26.742627 12265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:26.742846 12265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:26.745247 12265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "e61b62698edf4856972ac39c259cf2ff"
format_stamp: "Formatted at 2026-08-10 23:05:26 on dist-test-slave-6bbx"
server_key: "0c4548197d6d5b7f7cc1a5d591588933"
server_key_iv: "9ff630811925a08c353d1893b1533fe4"
server_key_version: "encryptionkey@0"
I20260810 23:05:26.745877 12265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "e61b62698edf4856972ac39c259cf2ff"
format_stamp: "Formatted at 2026-08-10 23:05:26 on dist-test-slave-6bbx"
server_key: "0c4548197d6d5b7f7cc1a5d591588933"
server_key_iv: "9ff630811925a08c353d1893b1533fe4"
server_key_version: "encryptionkey@0"
I20260810 23:05:26.750329 12265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:05:26.753335 12288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:26.754951 12265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:05:26.755090 12265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "e61b62698edf4856972ac39c259cf2ff"
format_stamp: "Formatted at 2026-08-10 23:05:26 on dist-test-slave-6bbx"
server_key: "0c4548197d6d5b7f7cc1a5d591588933"
server_key_iv: "9ff630811925a08c353d1893b1533fe4"
server_key_version: "encryptionkey@0"
I20260810 23:05:26.755239 12265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:26.790695 12265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:26.794116 12265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:26.794310 12265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:26.794975 12265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:26.795904 12265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:26.795957 12265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:26.796000 12265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:26.796022 12265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:26.805023 12265 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:44907
I20260810 23:05:26.805066 12460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:44907 every 8 connection(s)
I20260810 23:05:26.806061 12265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:26.807018 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 12265
I20260810 23:05:26.807132 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:05:26.807430 15584 external_mini_cluster.cc:1428] Setting key 266f62335747715556eb8fffbb72a319
Aug 10 23:05:26 dist-test-slave-6bbx krb5kdc[12103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403126, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
I20260810 23:05:26.822504 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:26.808188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:47489 (local address 127.15.56.62:43085)
0810 23:05:26.808336 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:26.808340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:26.809694 (+ 1354us) server_negotiation.cc:407] Connection header received
0810 23:05:26.810907 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:26.810911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:26.810969 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:26.811057 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:26.812839 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:26.813555 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:26.814799 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:26.815156 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:26.815280 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:26.817975 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:26.817980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:26.817983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:26.818017 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:26.819650 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:26.820391 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:26.820394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:26.820395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:26.820494 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:26.820946 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:26.820949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:26.820950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:26.821195 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:26.821292 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:26.822162 (+ 870us) server_negotiation.cc:299] Negotiation successful
0810 23:05:26.822333 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 23:05:26.823428 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:26.808652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43085 (local address 127.15.56.1:47489)
0810 23:05:26.809493 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:26.809560 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:26.810611 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:26.811302 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:26.811313 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:26.811911 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:26.812631 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:26.812654 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:26.813695 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:26.813698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:26.814669 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:26.814681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:26.815331 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:26.815334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:26.815523 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:26.816188 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:26.816206 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:26.817763 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:26.819826 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:26.819832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:26.819851 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:26.820217 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:26.820629 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:26.820632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:26.820634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:26.820826 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:26.821400 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:26.821408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:26.821762 (+ 354us) client_negotiation.cc:769] Sending connection context
0810 23:05:26.822034 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 23:05:26.822449 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":149,"threads_started":1}
I20260810 23:05:26.824971 12461 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43085
I20260810 23:05:26.825300 12461 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:26.825981 12461 heartbeater.cc:511] Master 127.15.56.62:43085 requested a full tablet report, sending...
I20260810 23:05:26.827736 12172 ts_manager.cc:194] Registered new tserver with Master: e61b62698edf4856972ac39c259cf2ff (127.15.56.1:44907)
I20260810 23:05:26.828980 12172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:47489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:05:26.843070 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43085
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:26.961777 12468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:26.962136 12468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:26.962194 12468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:26.966562 12468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:26.966639 12468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:26.966707 12468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:05:26.972481 12468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:26.973664 12468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:26.974737 12468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:05:26.984751 12468 server_base.cc:1034] running on GCE node
W20260810 23:05:26.984696 12473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:26.984687 12474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:26.984690 12476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:26.985562 12468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:26.986205 12468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:26.987376 12468 hybrid_clock.cc:648] HybridClock initialized: now 1786403126987345 us; error 43 us; skew 500 ppm
Aug 10 23:05:26 dist-test-slave-6bbx krb5kdc[12103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403126, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:26.990221 12468 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:05:26.991083 12468 webserver.cc:460] Webserver started at http://127.15.56.2:45841/ using document root <none> and password file <none>
I20260810 23:05:26.991618 12468 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:26.991722 12468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:26.991927 12468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:26.994009 12468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "d5150bc8e841447fb5d5d7dcec16d38c"
format_stamp: "Formatted at 2026-08-10 23:05:26 on dist-test-slave-6bbx"
server_key: "31665dd3be02f1a29213a47e2a3375f5"
server_key_iv: "7293be1ae1ee08af0131e58702f1615f"
server_key_version: "encryptionkey@0"
I20260810 23:05:26.994541 12468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "d5150bc8e841447fb5d5d7dcec16d38c"
format_stamp: "Formatted at 2026-08-10 23:05:26 on dist-test-slave-6bbx"
server_key: "31665dd3be02f1a29213a47e2a3375f5"
server_key_iv: "7293be1ae1ee08af0131e58702f1615f"
server_key_version: "encryptionkey@0"
I20260810 23:05:26.998394 12468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:05:27.000969 12488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:27.002327 12468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:05:27.002437 12468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "d5150bc8e841447fb5d5d7dcec16d38c"
format_stamp: "Formatted at 2026-08-10 23:05:26 on dist-test-slave-6bbx"
server_key: "31665dd3be02f1a29213a47e2a3375f5"
server_key_iv: "7293be1ae1ee08af0131e58702f1615f"
server_key_version: "encryptionkey@0"
I20260810 23:05:27.002534 12468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:27.021077 12468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:27.024685 12468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:27.024861 12468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:27.025476 12468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:27.026352 12468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:27.026402 12468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:27.026443 12468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:27.026463 12468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:27.035243 12468 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:36515
I20260810 23:05:27.035288 12648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:36515 every 8 connection(s)
I20260810 23:05:27.036244 12468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:27.041582 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 12468
I20260810 23:05:27.041707 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:05:27.042014 15584 external_mini_cluster.cc:1428] Setting key 1b4c77f99428db88b8398e5400195fdf
Aug 10 23:05:27 dist-test-slave-6bbx krb5kdc[12103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403126, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:05:27.051592 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.038181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:49687 (local address 127.15.56.62:43085)
0810 23:05:27.038319 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:27.038323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:27.039482 (+ 1159us) server_negotiation.cc:407] Connection header received
0810 23:05:27.040387 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:27.040390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:27.040440 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:27.040516 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:27.041735 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.042505 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.043962 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.044255 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.044360 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.047224 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:27.047228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:27.047230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:27.047256 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:27.048855 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:27.049619 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:27.049623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:27.049625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:27.049780 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:27.050219 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:27.050223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:27.050225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:27.050541 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:27.050663 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:27.051261 (+ 598us) server_negotiation.cc:299] Negotiation successful
0810 23:05:27.051414 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:05:27.052526 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.038587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43085 (local address 127.15.56.2:49687)
0810 23:05:27.039314 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:27.039359 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:27.040209 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:27.040687 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:27.040695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:27.041049 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:27.041543 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.041557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.042661 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.042665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.043830 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.043842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.044414 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.044417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.044575 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.045333 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:27.045352 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:27.047029 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:27.049030 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:27.049040 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:27.049064 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:27.049487 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:27.049918 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:27.049922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:27.049923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:27.050104 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:27.050689 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:27.050695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:27.050999 (+ 304us) client_negotiation.cc:769] Sending connection context
0810 23:05:27.051203 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 23:05:27.051522 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:05:27.054065 12650 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43085
I20260810 23:05:27.054369 12650 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:27.054951 12650 heartbeater.cc:511] Master 127.15.56.62:43085 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
I20260810 23:05:27.056252 12172 ts_manager.cc:194] Registered new tserver with Master: d5150bc8e841447fb5d5d7dcec16d38c (127.15.56.2:36515)
I20260810 23:05:27.056862 12172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:49687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:05:27.073053 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43085
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:27.191900 12657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:27.192199 12657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:27.192243 12657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:27.196344 12657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:27.196408 12657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:27.196467 12657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:05:27.201892 12657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:43085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:27.202955 12657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:27.204035 12657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:27.214094 12664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:27.214218 12663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:27.214131 12667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:27.214373 12657 server_base.cc:1034] running on GCE node
I20260810 23:05:27.214886 12657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:27.215701 12657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:27.216905 12657 hybrid_clock.cc:648] HybridClock initialized: now 1786403127216875 us; error 41 us; skew 500 ppm
Aug 10 23:05:27 dist-test-slave-6bbx krb5kdc[12103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403127, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:27.219992 12657 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:05:27.220994 12657 webserver.cc:460] Webserver started at http://127.15.56.3:44895/ using document root <none> and password file <none>
I20260810 23:05:27.221592 12657 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:27.221675 12657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:27.221903 12657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:27.224161 12657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "cd2d5dffd1a844338621e970ee432d77"
format_stamp: "Formatted at 2026-08-10 23:05:27 on dist-test-slave-6bbx"
server_key: "1ebfc7ac0fa76406badb3c1f8919d608"
server_key_iv: "a944097c896d4d35084c1e3e138b0553"
server_key_version: "encryptionkey@0"
I20260810 23:05:27.224743 12657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "cd2d5dffd1a844338621e970ee432d77"
format_stamp: "Formatted at 2026-08-10 23:05:27 on dist-test-slave-6bbx"
server_key: "1ebfc7ac0fa76406badb3c1f8919d608"
server_key_iv: "a944097c896d4d35084c1e3e138b0553"
server_key_version: "encryptionkey@0"
I20260810 23:05:27.229030 12657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:27.231873 12676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:27.233189 12657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:27.233335 12657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "cd2d5dffd1a844338621e970ee432d77"
format_stamp: "Formatted at 2026-08-10 23:05:27 on dist-test-slave-6bbx"
server_key: "1ebfc7ac0fa76406badb3c1f8919d608"
server_key_iv: "a944097c896d4d35084c1e3e138b0553"
server_key_version: "encryptionkey@0"
I20260810 23:05:27.233444 12657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:27.249128 12657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:27.253063 12657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:27.253247 12657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:27.253875 12657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:27.254753 12657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:27.254806 12657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:27.254849 12657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:27.254871 12657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:27.264214 12657 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:46289
I20260810 23:05:27.264237 12847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:46289 every 8 connection(s)
I20260810 23:05:27.265252 12657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:05:27.271831 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 12657
I20260810 23:05:27.271991 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:05:27.272367 15584 external_mini_cluster.cc:1428] Setting key 3495ed86258d4e2c90f11635a333fc22
Aug 10 23:05:27 dist-test-slave-6bbx krb5kdc[12103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403127, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:05:27.279841 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.267341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:43883 (local address 127.15.56.62:43085)
0810 23:05:27.267510 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:27.267515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:27.268511 (+ 996us) server_negotiation.cc:407] Connection header received
0810 23:05:27.269796 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:27.269800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:27.269851 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:27.269926 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:27.271131 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.271692 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.273178 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.273480 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.273574 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.276251 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:27.276255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:27.276257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:27.276286 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:27.277583 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:27.278140 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:27.278143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:27.278144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:27.278243 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:27.278596 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:27.278599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:27.278600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:27.278846 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:27.278938 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:27.279499 (+ 561us) server_negotiation.cc:299] Negotiation successful
0810 23:05:27.279627 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:05:27.280738 12850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.267850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43085 (local address 127.15.56.3:43883)
0810 23:05:27.268351 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:27.268398 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:27.269565 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:27.270084 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:27.270091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:27.270487 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:27.271016 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.271030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.271837 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.271841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.273044 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.273057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.273635 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.273639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.273836 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.274448 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:27.274472 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:27.276041 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:27.277708 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:27.277714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:27.277731 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:27.278023 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:27.278328 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:27.278330 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:27.278332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:27.278483 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:27.278944 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:27.278949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:27.279229 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 23:05:27.279408 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 23:05:27.279755 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260810 23:05:27.282122 12848 heartbeater.cc:347] Connected to a master server at 127.15.56.62:43085
I20260810 23:05:27.282411 12848 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:27.283005 12848 heartbeater.cc:511] Master 127.15.56.62:43085 requested a full tablet report, sending...
I20260810 23:05:27.284216 12172 ts_manager.cc:194] Registered new tserver with Master: cd2d5dffd1a844338621e970ee432d77 (127.15.56.3:46289)
I20260810 23:05:27.284819 12172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:43883
I20260810 23:05:27.286510 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:05:27.296317 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.288514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.15.56.62:43085)
0810 23:05:27.288739 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:27.288744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:27.288868 (+ 124us) server_negotiation.cc:407] Connection header received
0810 23:05:27.289010 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:27.289014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:27.289057 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:27.289119 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:27.289905 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.290425 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.291289 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.291511 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.291600 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.293324 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:27.293327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:27.293329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:27.293354 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:27.294484 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:27.294997 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:27.295000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:27.295001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:27.295077 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:27.295391 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:27.295394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:27.295395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:27.295607 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:27.295813 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:27.296064 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 23:05:27.296161 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260810 23:05:27.299290 12172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:05:27.302563 12172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:05:27.318018 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.312150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.15.56.1:44907)
0810 23:05:27.312759 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:27.312786 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:27.312819 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:05:27.313040 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:27.313047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:27.313302 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:27.313454 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:27.313780 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.314945 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.316421 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.317177 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.317244 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.317364 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:27.317768 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 23:05:27.317865 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":85,"threads_started":1}
I20260810 23:05:27.318009 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.311888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:36515 (local address 127.0.0.1:47396)
0810 23:05:27.312950 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:27.312962 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:27.313054 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:27.313889 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:27.313892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:27.313911 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:27.314008 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.314014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.315261 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.315265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.316474 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.316483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.317550 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.317553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.317643 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.317663 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 23:05:27.317718 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 23:05:27.317849 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"mutex_wait_us":183,"thread_start_us":152,"threads_started":1}
I20260810 23:05:27.318019 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.312150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.15.56.2:36515)
0810 23:05:27.313312 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:27.313326 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:27.313349 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:05:27.313423 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:27.313428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:27.313682 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:27.313834 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:27.314130 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.315115 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.316600 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.317427 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.317542 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.317707 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:27.317762 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 23:05:27.317859 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050,"thread_start_us":153,"threads_started":1}
I20260810 23:05:27.318398 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.311889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:44907 (local address 127.0.0.1:45504)
0810 23:05:27.312630 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:27.312674 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:27.312823 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:27.313456 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:27.313462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:27.313485 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:27.313621 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.313630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.315070 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.315072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.316288 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.316298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.317444 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.317446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.317541 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.317584 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 23:05:27.318196 (+ 612us) client_negotiation.cc:240] Negotiation successful
0810 23:05:27.318271 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":109,"threads_started":1}
I20260810 23:05:27.319425 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.312739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:46289 (local address 127.0.0.1:59046)
0810 23:05:27.314044 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:27.314055 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:27.314162 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:27.314679 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:27.314682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:27.314731 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:27.314818 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.314824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.316344 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.316348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.317443 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.317453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.319043 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.319047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.319163 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.319189 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:05:27.319252 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:05:27.319304 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"thread_start_us":80,"threads_started":1}
I20260810 23:05:27.320393 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.312939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.15.56.3:46289)
0810 23:05:27.313809 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:27.313822 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:27.314147 (+ 325us) server_negotiation.cc:407] Connection header received
0810 23:05:27.314279 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:27.314285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:27.314529 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:27.314663 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:27.314928 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.316201 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.317571 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.318910 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.319024 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.320093 (+ 1069us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:27.320162 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 23:05:27.320263 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":162,"threads_started":1}
I20260810 23:05:27.321437 12365 tablet_service.cc:1467] Processing CreateTablet for tablet b180eb879ec242f4ac6cd9b863fd4fe2 (DEFAULT_TABLE table=test-table [id=44391b6a58ca4940be6c1fd523552a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:27.321439 12558 tablet_service.cc:1467] Processing CreateTablet for tablet b180eb879ec242f4ac6cd9b863fd4fe2 (DEFAULT_TABLE table=test-table [id=44391b6a58ca4940be6c1fd523552a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:27.322701 12365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b180eb879ec242f4ac6cd9b863fd4fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:27.322705 12558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b180eb879ec242f4ac6cd9b863fd4fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:27.322914 12733 tablet_service.cc:1467] Processing CreateTablet for tablet b180eb879ec242f4ac6cd9b863fd4fe2 (DEFAULT_TABLE table=test-table [id=44391b6a58ca4940be6c1fd523552a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:27.324005 12733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b180eb879ec242f4ac6cd9b863fd4fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:27.330163 12876 tablet_bootstrap.cc:492] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77: Bootstrap starting.
I20260810 23:05:27.331326 12877 tablet_bootstrap.cc:492] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff: Bootstrap starting.
I20260810 23:05:27.331910 12879 tablet_bootstrap.cc:492] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c: Bootstrap starting.
I20260810 23:05:27.332561 12876 tablet_bootstrap.cc:654] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:27.333621 12877 tablet_bootstrap.cc:654] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:27.333665 12876 log.cc:824] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:27.333999 12879 tablet_bootstrap.cc:654] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:27.334514 12877 log.cc:824] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:27.335218 12879 log.cc:824] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:27.336213 12876 tablet_bootstrap.cc:492] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77: No bootstrap required, opened a new log
I20260810 23:05:27.336431 12876 ts_tablet_manager.cc:1397] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:05:27.336598 12877 tablet_bootstrap.cc:492] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff: No bootstrap required, opened a new log
I20260810 23:05:27.336809 12877 ts_tablet_manager.cc:1397] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:05:27.337390 12879 tablet_bootstrap.cc:492] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c: No bootstrap required, opened a new log
I20260810 23:05:27.337586 12879 ts_tablet_manager.cc:1397] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:05:27.339936 12876 raft_consensus.cc:348] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61b62698edf4856972ac39c259cf2ff" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 44907 } } peers { permanent_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36515 } } peers { permanent_uuid: "cd2d5dffd1a844338621e970ee432d77" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 46289 } }
I20260810 23:05:27.340232 12876 raft_consensus.cc:374] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:27.340292 12876 raft_consensus.cc:729] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2d5dffd1a844338621e970ee432d77, State: Initialized, Role: FOLLOWER
I20260810 23:05:27.340780 12876 consensus_queue.cc:260] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61b62698edf4856972ac39c259cf2ff" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 44907 } } peers { permanent_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36515 } } peers { permanent_uuid: "cd2d5dffd1a844338621e970ee432d77" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 46289 } }
I20260810 23:05:27.341559 12848 heartbeater.cc:503] Master 127.15.56.62:43085 was elected leader, sending a full tablet report...
I20260810 23:05:27.341670 12877 raft_consensus.cc:348] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61b62698edf4856972ac39c259cf2ff" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 44907 } } peers { permanent_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36515 } } peers { permanent_uuid: "cd2d5dffd1a844338621e970ee432d77" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 46289 } }
I20260810 23:05:27.341714 12879 raft_consensus.cc:348] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61b62698edf4856972ac39c259cf2ff" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 44907 } } peers { permanent_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36515 } } peers { permanent_uuid: "cd2d5dffd1a844338621e970ee432d77" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 46289 } }
I20260810 23:05:27.341985 12877 raft_consensus.cc:374] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:27.341985 12879 raft_consensus.cc:374] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:27.342037 12877 raft_consensus.cc:729] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61b62698edf4856972ac39c259cf2ff, State: Initialized, Role: FOLLOWER
I20260810 23:05:27.342037 12879 raft_consensus.cc:729] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5150bc8e841447fb5d5d7dcec16d38c, State: Initialized, Role: FOLLOWER
I20260810 23:05:27.342594 12879 consensus_queue.cc:260] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61b62698edf4856972ac39c259cf2ff" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 44907 } } peers { permanent_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36515 } } peers { permanent_uuid: "cd2d5dffd1a844338621e970ee432d77" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 46289 } }
I20260810 23:05:27.342594 12877 consensus_queue.cc:260] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61b62698edf4856972ac39c259cf2ff" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 44907 } } peers { permanent_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36515 } } peers { permanent_uuid: "cd2d5dffd1a844338621e970ee432d77" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 46289 } }
I20260810 23:05:27.343788 12879 ts_tablet_manager.cc:1428] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:05:27.343788 12877 ts_tablet_manager.cc:1428] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 23:05:27.344175 12650 heartbeater.cc:503] Master 127.15.56.62:43085 was elected leader, sending a full tablet report...
I20260810 23:05:27.344463 12876 ts_tablet_manager.cc:1428] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 23:05:27.345175 12461 heartbeater.cc:503] Master 127.15.56.62:43085 was elected leader, sending a full tablet report...
I20260810 23:05:27.431100 12886 raft_consensus.cc:482] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:27.431402 12886 raft_consensus.cc:504] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61b62698edf4856972ac39c259cf2ff" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 44907 } } peers { permanent_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36515 } } peers { permanent_uuid: "cd2d5dffd1a844338621e970ee432d77" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 46289 } }
I20260810 23:05:27.432680 12886 leader_election.cc:288] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5150bc8e841447fb5d5d7dcec16d38c (127.15.56.2:36515), cd2d5dffd1a844338621e970ee432d77 (127.15.56.3:46289)
I20260810 23:05:27.437390 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.433135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:53875 (local address 127.15.56.2:36515)
0810 23:05:27.433279 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:27.433283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:27.433486 (+ 203us) server_negotiation.cc:407] Connection header received
0810 23:05:27.433720 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:27.433724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:27.433784 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:27.433870 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:27.434234 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.435013 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.436332 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.436839 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.436942 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.437013 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:27.437183 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 23:05:27.437261 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:05:27.437405 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.433044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:36515 (local address 127.15.56.1:53875)
0810 23:05:27.433391 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:27.433406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:27.433532 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:27.433910 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:27.433913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:27.433938 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:27.434081 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.434090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.435134 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.435137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.436179 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.436191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.436968 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.436972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.437068 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.437089 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:05:27.437157 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:05:27.437237 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":141,"threads_started":1}
I20260810 23:05:27.437564 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.433677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:52577 (local address 127.15.56.3:46289)
0810 23:05:27.433829 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:27.433832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:27.434125 (+ 293us) server_negotiation.cc:407] Connection header received
0810 23:05:27.434246 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:27.434250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:27.434304 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:27.434391 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:27.434608 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.435254 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.436557 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.436982 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.437084 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.437164 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:27.437343 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 23:05:27.437421 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:05:27.437598 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.433523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:46289 (local address 127.15.56.1:52577)
0810 23:05:27.434002 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:27.434015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:27.434127 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:27.434399 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:27.434401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:27.434416 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:27.434509 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.434514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.435378 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.435382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.436430 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:27.436442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.437108 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.437111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:27.437222 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.437246 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:05:27.437316 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:05:27.437389 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260810 23:05:27.438232 12582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b180eb879ec242f4ac6cd9b863fd4fe2" candidate_uuid: "e61b62698edf4856972ac39c259cf2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" is_pre_election: true
I20260810 23:05:27.438220 12784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b180eb879ec242f4ac6cd9b863fd4fe2" candidate_uuid: "e61b62698edf4856972ac39c259cf2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2d5dffd1a844338621e970ee432d77" is_pre_election: true
I20260810 23:05:27.438558 12784 raft_consensus.cc:2454] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e61b62698edf4856972ac39c259cf2ff in term 0.
I20260810 23:05:27.438561 12582 raft_consensus.cc:2454] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e61b62698edf4856972ac39c259cf2ff in term 0.
I20260810 23:05:27.439087 12289 leader_election.cc:302] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd2d5dffd1a844338621e970ee432d77, e61b62698edf4856972ac39c259cf2ff; no voters:
I20260810 23:05:27.439388 12886 raft_consensus.cc:2781] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:05:27.439474 12886 raft_consensus.cc:482] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:05:27.439503 12886 raft_consensus.cc:3037] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:27.440614 12886 raft_consensus.cc:504] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61b62698edf4856972ac39c259cf2ff" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 44907 } } peers { permanent_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36515 } } peers { permanent_uuid: "cd2d5dffd1a844338621e970ee432d77" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 46289 } }
I20260810 23:05:27.440972 12886 leader_election.cc:288] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [CANDIDATE]: Term 1 election: Requested vote from peers d5150bc8e841447fb5d5d7dcec16d38c (127.15.56.2:36515), cd2d5dffd1a844338621e970ee432d77 (127.15.56.3:46289)
I20260810 23:05:27.441403 12582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b180eb879ec242f4ac6cd9b863fd4fe2" candidate_uuid: "e61b62698edf4856972ac39c259cf2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5150bc8e841447fb5d5d7dcec16d38c"
I20260810 23:05:27.441510 12784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b180eb879ec242f4ac6cd9b863fd4fe2" candidate_uuid: "e61b62698edf4856972ac39c259cf2ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2d5dffd1a844338621e970ee432d77"
I20260810 23:05:27.441527 12582 raft_consensus.cc:3037] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:27.441643 12784 raft_consensus.cc:3037] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:27.442657 12582 raft_consensus.cc:2454] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61b62698edf4856972ac39c259cf2ff in term 1.
I20260810 23:05:27.442727 12784 raft_consensus.cc:2454] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61b62698edf4856972ac39c259cf2ff in term 1.
I20260810 23:05:27.443068 12289 leader_election.cc:302] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd2d5dffd1a844338621e970ee432d77, e61b62698edf4856972ac39c259cf2ff; no voters:
I20260810 23:05:27.443244 12886 raft_consensus.cc:2781] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:27.443470 12886 raft_consensus.cc:686] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [term 1 LEADER]: Becoming Leader. State: Replica: e61b62698edf4856972ac39c259cf2ff, State: Running, Role: LEADER
I20260810 23:05:27.443850 12886 consensus_queue.cc:237] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61b62698edf4856972ac39c259cf2ff" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 44907 } } peers { permanent_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36515 } } peers { permanent_uuid: "cd2d5dffd1a844338621e970ee432d77" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 46289 } }
I20260810 23:05:27.447438 12172 catalog_manager.cc:5495] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff reported cstate change: term changed from 0 to 1, leader changed from <none> to e61b62698edf4856972ac39c259cf2ff (127.15.56.1). New cstate: current_term: 1 leader_uuid: "e61b62698edf4856972ac39c259cf2ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61b62698edf4856972ac39c259cf2ff" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 44907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd2d5dffd1a844338621e970ee432d77" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 46289 } health_report { overall_health: UNKNOWN } } }
I20260810 23:05:27.475749 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.472173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.15.56.1:44907)
0810 23:05:27.472336 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:27.472339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:27.472376 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:05:27.472455 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:27.472457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:27.472525 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:27.472610 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:05:27.472826 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.473327 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.474185 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.474417 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.474510 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.474772 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:05:27.475215 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:05:27.475333 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:27.475501 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 23:05:27.475552 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:05:27.482860 12784 raft_consensus.cc:1261] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77 [term 1 FOLLOWER]: Refusing update from remote peer e61b62698edf4856972ac39c259cf2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:27.482890 12582 raft_consensus.cc:1261] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c [term 1 FOLLOWER]: Refusing update from remote peer e61b62698edf4856972ac39c259cf2ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:27.483732 12886 consensus_queue.cc:1035] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5150bc8e841447fb5d5d7dcec16d38c" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 36515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:27.483994 12889 consensus_queue.cc:1035] T b180eb879ec242f4ac6cd9b863fd4fe2 P e61b62698edf4856972ac39c259cf2ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd2d5dffd1a844338621e970ee432d77" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 46289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:27.492666 12892 mvcc.cc:205] Tried to move back new op lower bound from 7317107210160766976 to 7317107210012966912. Current Snapshot: MvccSnapshot[applied={T|T < 7317107210160766976}]
I20260810 23:05:27.495887 12895 mvcc.cc:205] Tried to move back new op lower bound from 7317107210160766976 to 7317107210012966912. Current Snapshot: MvccSnapshot[applied={T|T < 7317107210160766976}]
I20260810 23:05:27.502502 12169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:27.502686 12169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:27.505334 12169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37c744d70e4c4d0bb558149e8e062c3a: Sending DeleteTablet for 3 replicas of tablet b180eb879ec242f4ac6cd9b863fd4fe2
I20260810 23:05:27.506688 12365 tablet_service.cc:1514] Processing DeleteTablet for tablet b180eb879ec242f4ac6cd9b863fd4fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:27 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:45504
I20260810 23:05:27.507211 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 12265
I20260810 23:05:27.507391 12558 tablet_service.cc:1514] Processing DeleteTablet for tablet b180eb879ec242f4ac6cd9b863fd4fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:27 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:47396
I20260810 23:05:27.510609 12733 tablet_service.cc:1514] Processing DeleteTablet for tablet b180eb879ec242f4ac6cd9b863fd4fe2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:27 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:59046
I20260810 23:05:27.512804 12912 tablet_replica.cc:330] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77: stopping tablet replica
I20260810 23:05:27.513391 12912 raft_consensus.cc:2229] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:27.513942 12912 raft_consensus.cc:2258] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:27.517526 12911 tablet_replica.cc:330] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c: stopping tablet replica
I20260810 23:05:27.517948 12911 raft_consensus.cc:2229] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:27.518394 12911 raft_consensus.cc:2258] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:27.519621 12912 ts_tablet_manager.cc:1905] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:27.520010 12911 ts_tablet_manager.cc:1905] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:27.522673 12911 ts_tablet_manager.cc:1918] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:27.522673 12912 ts_tablet_manager.cc:1918] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:27.522804 12912 log.cc:1196] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/b180eb879ec242f4ac6cd9b863fd4fe2
I20260810 23:05:27.522804 12911 log.cc:1196] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/b180eb879ec242f4ac6cd9b863fd4fe2
I20260810 23:05:27.523135 12912 ts_tablet_manager.cc:1939] T b180eb879ec242f4ac6cd9b863fd4fe2 P cd2d5dffd1a844338621e970ee432d77: Deleting consensus metadata
I20260810 23:05:27.523135 12911 ts_tablet_manager.cc:1939] T b180eb879ec242f4ac6cd9b863fd4fe2 P d5150bc8e841447fb5d5d7dcec16d38c: Deleting consensus metadata
W20260810 23:05:27.524515 12151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:44907 (error 108)
I20260810 23:05:27.524686 12147 catalog_manager.cc:4841] TS cd2d5dffd1a844338621e970ee432d77 (127.15.56.3:46289): tablet b180eb879ec242f4ac6cd9b863fd4fe2 (table test-table [id=44391b6a58ca4940be6c1fd523552a27]) successfully deleted
I20260810 23:05:27.524938 12147 catalog_manager.cc:4841] TS d5150bc8e841447fb5d5d7dcec16d38c (127.15.56.2:36515): tablet b180eb879ec242f4ac6cd9b863fd4fe2 (table test-table [id=44391b6a58ca4940be6c1fd523552a27]) successfully deleted
I20260810 23:05:27.525579 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 12468
I20260810 23:05:27.525843 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.525544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:44907 (local address 127.0.0.1:45518)
0810 23:05:27.525683 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:27.525753 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:05:27.528764 12151 catalog_manager.cc:4584] TS e61b62698edf4856972ac39c259cf2ff (127.15.56.1:44907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b180eb879ec242f4ac6cd9b863fd4fe2: Network error: Client connection negotiation failed: client connection to 127.15.56.1:44907: connect: Connection refused (error 111)
I20260810 23:05:27.541719 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 12657
I20260810 23:05:27.558543 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 12121
2026-08-10T23:05:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2584 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T23:05:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:27Z Disabled control of system clock
I20260810 23:05:27.606029 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33179
--webserver_interface=127.15.56.62
--webserver_port=0
--builtin_ntp_servers=127.15.56.20:41069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:33179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:05:27.725025 12921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:27.725350 12921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:27.725390 12921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:27.729579 12921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:27.729664 12921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:27.729679 12921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:27.729694 12921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:27.729708 12921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:27.734854 12921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:41069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:33179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:33179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:27.736235 12921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:27.737859 12921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:27.747319 12931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:27.747344 12927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:27.747406 12921 server_base.cc:1034] running on GCE node
W20260810 23:05:27.747345 12928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:27.748332 12921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:27.749691 12921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:27.750907 12921 hybrid_clock.cc:648] HybridClock initialized: now 1786403127750885 us; error 40 us; skew 500 ppm
I20260810 23:05:27.753175 12921 webserver.cc:460] Webserver started at http://127.15.56.62:45363/ using document root <none> and password file <none>
I20260810 23:05:27.753824 12921 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:27.753906 12921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:27.754169 12921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:27.756553 12921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "9ef0f9fa18df47a2931289cd0cfc55d6"
format_stamp: "Formatted at 2026-08-10 23:05:27 on dist-test-slave-6bbx"
server_key: "925223692811d4144c010ebf7dd35071"
server_key_iv: "9c75050b08cc6e60b59b440cc959b1ab"
server_key_version: "encryptionkey@0"
I20260810 23:05:27.757202 12921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "9ef0f9fa18df47a2931289cd0cfc55d6"
format_stamp: "Formatted at 2026-08-10 23:05:27 on dist-test-slave-6bbx"
server_key: "925223692811d4144c010ebf7dd35071"
server_key_iv: "9c75050b08cc6e60b59b440cc959b1ab"
server_key_version: "encryptionkey@0"
I20260810 23:05:27.761612 12921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 23:05:27.764941 12942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:27.766477 12921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:27.766641 12921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "9ef0f9fa18df47a2931289cd0cfc55d6"
format_stamp: "Formatted at 2026-08-10 23:05:27 on dist-test-slave-6bbx"
server_key: "925223692811d4144c010ebf7dd35071"
server_key_iv: "9c75050b08cc6e60b59b440cc959b1ab"
server_key_version: "encryptionkey@0"
I20260810 23:05:27.766774 12921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:27.781556 12921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:27.782375 12921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:27.782572 12921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:27.789719 12921 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:33179
I20260810 23:05:27.789774 13020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:33179 every 8 connection(s)
I20260810 23:05:27.790869 12921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:27.793247 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 12921
I20260810 23:05:27.793381 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:27.793690 15584 external_mini_cluster.cc:1428] Setting key b8780943023bfe3e662b249557f97a5b
I20260810 23:05:27.795682 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:27.802484 13021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6: Bootstrap starting.
I20260810 23:05:27.803864 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:27.795275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.15.56.62:33179)
0810 23:05:27.796053 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:27.796071 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:27.796111 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:05:27.796809 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:27.796827 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:27.797278 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:27.797582 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:27.797949 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.798948 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.800207 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:27.800766 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:27.800911 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:27.801888 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:27.801894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:27.801909 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:27.802368 (+ 459us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:27.802431 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:27.802453 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:27.802536 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:27.802740 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 23:05:27.803037 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":138,"threads_started":1}
I20260810 23:05:27.805048 13021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:27.805966 13021 log.cc:824] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:27.808037 13021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6: No bootstrap required, opened a new log
I20260810 23:05:27.811828 13021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef0f9fa18df47a2931289cd0cfc55d6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33179 } }
I20260810 23:05:27.812117 13021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:27.812158 13021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef0f9fa18df47a2931289cd0cfc55d6, State: Initialized, Role: FOLLOWER
I20260810 23:05:27.812685 13021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef0f9fa18df47a2931289cd0cfc55d6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33179 } }
I20260810 23:05:27.812844 13021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:27.812889 13021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:27.812973 13021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:27.813932 13021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef0f9fa18df47a2931289cd0cfc55d6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33179 } }
I20260810 23:05:27.814273 13021 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef0f9fa18df47a2931289cd0cfc55d6; no voters:
I20260810 23:05:27.814602 13021 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:27.814742 13026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:27.814985 13026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [term 1 LEADER]: Becoming Leader. State: Replica: 9ef0f9fa18df47a2931289cd0cfc55d6, State: Running, Role: LEADER
I20260810 23:05:27.815302 13026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef0f9fa18df47a2931289cd0cfc55d6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33179 } }
I20260810 23:05:27.815905 13021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:27.817124 13027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef0f9fa18df47a2931289cd0cfc55d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef0f9fa18df47a2931289cd0cfc55d6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33179 } } }
I20260810 23:05:27.817149 13028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef0f9fa18df47a2931289cd0cfc55d6. Latest consensus state: current_term: 1 leader_uuid: "9ef0f9fa18df47a2931289cd0cfc55d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef0f9fa18df47a2931289cd0cfc55d6" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 33179 } } }
I20260810 23:05:27.817380 13028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:27.817308 13027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:27.817920 13042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:27.821333 13042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:27.827950 13042 catalog_manager.cc:1347] Generated new cluster ID: daa4c9cda6c04602a2e53c2982c0d465
I20260810 23:05:27.828033 13042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:27.836164 13042 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:27.837275 13042 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:27.842803 13042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6: Generated new TSK 0
I20260810 23:05:27.843580 13042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:27.853500 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33179
--builtin_ntp_servers=127.15.56.20:41069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:05:27.971980 13057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:27.972405 13057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:27.972461 13057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:27.976637 13057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:27.976709 13057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:27.976768 13057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:05:27.981791 13057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:41069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:27.983053 13057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:27.984743 13057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:27.995302 13069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:27.995306 13067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:27.995302 13066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:27.995419 13057 server_base.cc:1034] running on GCE node
I20260810 23:05:27.996071 13057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:27.996778 13057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:27.997948 13057 hybrid_clock.cc:648] HybridClock initialized: now 1786403127997914 us; error 48 us; skew 500 ppm
I20260810 23:05:27.999969 13057 webserver.cc:460] Webserver started at http://127.15.56.1:40949/ using document root <none> and password file <none>
I20260810 23:05:28.000604 13057 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:28.000689 13057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:28.000916 13057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:28.003140 13057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "9ead495ad7c446eea92d679a9f30dd88"
format_stamp: "Formatted at 2026-08-10 23:05:27 on dist-test-slave-6bbx"
server_key: "c1d37c47938b2fabce093515c85d1630"
server_key_iv: "2cc782fcd93559c76fbb175a64d321f1"
server_key_version: "encryptionkey@0"
I20260810 23:05:28.003803 13057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "9ead495ad7c446eea92d679a9f30dd88"
format_stamp: "Formatted at 2026-08-10 23:05:27 on dist-test-slave-6bbx"
server_key: "c1d37c47938b2fabce093515c85d1630"
server_key_iv: "2cc782fcd93559c76fbb175a64d321f1"
server_key_version: "encryptionkey@0"
I20260810 23:05:28.007777 13057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:28.010617 13076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:28.012342 13057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:28.012496 13057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "9ead495ad7c446eea92d679a9f30dd88"
format_stamp: "Formatted at 2026-08-10 23:05:27 on dist-test-slave-6bbx"
server_key: "c1d37c47938b2fabce093515c85d1630"
server_key_iv: "2cc782fcd93559c76fbb175a64d321f1"
server_key_version: "encryptionkey@0"
I20260810 23:05:28.012632 13057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:28.032063 13057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:28.032820 13057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:28.032989 13057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:28.033609 13057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:28.034804 13057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:28.034873 13057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:28.034926 13057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:28.034955 13057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:28.044202 13057 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:46131
I20260810 23:05:28.044235 13268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:46131 every 8 connection(s)
I20260810 23:05:28.045192 13057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:28.051653 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 13057
I20260810 23:05:28.051851 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:05:28.052145 15584 external_mini_cluster.cc:1428] Setting key ebf9566db9a10581e4231f3fe2773c1a
I20260810 23:05:28.054456 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33179
--builtin_ntp_servers=127.15.56.20:41069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:05:28.055931 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.047148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:49661 (local address 127.15.56.62:33179)
0810 23:05:28.047312 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:28.047316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:28.048552 (+ 1236us) server_negotiation.cc:407] Connection header received
0810 23:05:28.050062 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:28.050065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:28.050149 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:28.050236 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:28.051333 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.052044 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.053529 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.053855 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.053948 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.055376 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:28.055379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:28.055381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:28.055394 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:28.055416 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:28.055424 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:28.055493 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:28.055734 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 23:05:28.055810 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:05:28.057089 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.047596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33179 (local address 127.15.56.1:49661)
0810 23:05:28.048391 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:28.048438 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:28.049799 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:28.050467 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:28.050475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:28.050576 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:28.051119 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.051135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.052200 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.052204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.053387 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.053398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.054028 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.054032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.054257 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.055011 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:05:28.055013 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:28.055077 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:05:28.055182 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:28.055508 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:28.055517 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:28.055523 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:05:28.055652 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 23:05:28.056026 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":125,"threads_started":1}
I20260810 23:05:28.058713 13271 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33179
I20260810 23:05:28.059000 13271 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:28.059758 13271 heartbeater.cc:511] Master 127.15.56.62:33179 requested a full tablet report, sending...
I20260810 23:05:28.061565 12967 ts_manager.cc:194] Registered new tserver with Master: 9ead495ad7c446eea92d679a9f30dd88 (127.15.56.1:46131)
I20260810 23:05:28.062825 12967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.1:49661
W20260810 23:05:28.177186 13274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:28.177469 13274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:28.177510 13274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:28.181576 13274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:28.181625 13274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:28.181677 13274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:05:28.186435 13274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:41069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:28.187644 13274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:28.189245 13274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:28.199649 13281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:28.199733 13274 server_base.cc:1034] running on GCE node
W20260810 23:05:28.199699 13284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:28.199689 13282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:28.200294 13274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:28.200951 13274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:28.202154 13274 hybrid_clock.cc:648] HybridClock initialized: now 1786403128202129 us; error 32 us; skew 500 ppm
I20260810 23:05:28.204207 13274 webserver.cc:460] Webserver started at http://127.15.56.2:43455/ using document root <none> and password file <none>
I20260810 23:05:28.204805 13274 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:28.204874 13274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:28.205112 13274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:28.207273 13274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "f79adc951d814fc3aa03bb0c5b6c1b70"
format_stamp: "Formatted at 2026-08-10 23:05:28 on dist-test-slave-6bbx"
server_key: "20d2ec6e8860e173f8a82784c5711582"
server_key_iv: "ca2c3875fb142b153f9829256263b657"
server_key_version: "encryptionkey@0"
I20260810 23:05:28.207881 13274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "f79adc951d814fc3aa03bb0c5b6c1b70"
format_stamp: "Formatted at 2026-08-10 23:05:28 on dist-test-slave-6bbx"
server_key: "20d2ec6e8860e173f8a82784c5711582"
server_key_iv: "ca2c3875fb142b153f9829256263b657"
server_key_version: "encryptionkey@0"
I20260810 23:05:28.211951 13274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:28.214709 13294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:28.216305 13274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:28.216437 13274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "f79adc951d814fc3aa03bb0c5b6c1b70"
format_stamp: "Formatted at 2026-08-10 23:05:28 on dist-test-slave-6bbx"
server_key: "20d2ec6e8860e173f8a82784c5711582"
server_key_iv: "ca2c3875fb142b153f9829256263b657"
server_key_version: "encryptionkey@0"
I20260810 23:05:28.216575 13274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:28.236712 13274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:28.237458 13274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:28.237622 13274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:28.238229 13274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:28.239212 13274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:28.239260 13274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:28.239301 13274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:28.239318 13274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:28.248490 13274 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:34313
I20260810 23:05:28.248520 13466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:34313 every 8 connection(s)
I20260810 23:05:28.249558 13274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:28.253674 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 13274
I20260810 23:05:28.253793 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:05:28.254071 15584 external_mini_cluster.cc:1428] Setting key 0af8c644a24acb59d2820daeef5b3fa8
I20260810 23:05:28.256187 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33179
--builtin_ntp_servers=127.15.56.20:41069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:05:28.260144 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.251759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:41221 (local address 127.15.56.62:33179)
0810 23:05:28.251959 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:28.251962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:28.253250 (+ 1288us) server_negotiation.cc:407] Connection header received
0810 23:05:28.254852 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:28.254855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:28.254916 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:28.254995 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:28.256068 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.256653 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.257879 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.258117 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.258219 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.259531 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:28.259534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:28.259537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:28.259552 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:28.259575 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:28.259584 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:28.259694 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:28.259928 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 23:05:28.260012 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 23:05:28.261427 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.252210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33179 (local address 127.15.56.2:41221)
0810 23:05:28.253046 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:28.253094 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:28.254601 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:28.255213 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:28.255221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:28.255321 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:28.255920 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.255936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.256786 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.256789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.257749 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.257760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.258286 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.258290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.258469 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.259162 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:05:28.259164 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:28.259221 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:05:28.259322 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:28.259725 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:28.259737 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:28.259744 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:05:28.259885 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 23:05:28.260228 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":140,"threads_started":1}
I20260810 23:05:28.262984 13468 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33179
I20260810 23:05:28.263265 13468 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:28.263960 13468 heartbeater.cc:511] Master 127.15.56.62:33179 requested a full tablet report, sending...
I20260810 23:05:28.265390 12967 ts_manager.cc:194] Registered new tserver with Master: f79adc951d814fc3aa03bb0c5b6c1b70 (127.15.56.2:34313)
I20260810 23:05:28.266024 12967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.2:41221
W20260810 23:05:28.378862 13473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:28.379187 13473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:28.379247 13473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:28.383724 13473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:28.383821 13473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:28.383904 13473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:05:28.389134 13473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:41069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:33179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:28.390377 13473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:28.392057 13473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:28.402639 13488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:28.402661 13473 server_base.cc:1034] running on GCE node
W20260810 23:05:28.402642 13482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:28.402639 13484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:28.403362 13473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:28.404083 13473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:28.405282 13473 hybrid_clock.cc:648] HybridClock initialized: now 1786403128405260 us; error 31 us; skew 500 ppm
I20260810 23:05:28.407115 13473 webserver.cc:460] Webserver started at http://127.15.56.3:44993/ using document root <none> and password file <none>
I20260810 23:05:28.407730 13473 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:28.407804 13473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:28.407991 13473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:28.410053 13473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "a6e071bc9dee4f139d3d4e42aa758cb0"
format_stamp: "Formatted at 2026-08-10 23:05:28 on dist-test-slave-6bbx"
server_key: "2e308fc720c9956ed87f5f0accff9d8a"
server_key_iv: "5b9d3531bf7cff3c237029f352f5d6b9"
server_key_version: "encryptionkey@0"
I20260810 23:05:28.410574 13473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "a6e071bc9dee4f139d3d4e42aa758cb0"
format_stamp: "Formatted at 2026-08-10 23:05:28 on dist-test-slave-6bbx"
server_key: "2e308fc720c9956ed87f5f0accff9d8a"
server_key_iv: "5b9d3531bf7cff3c237029f352f5d6b9"
server_key_version: "encryptionkey@0"
I20260810 23:05:28.414258 13473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:05:28.416954 13498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:28.418377 13473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:05:28.418481 13473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "a6e071bc9dee4f139d3d4e42aa758cb0"
format_stamp: "Formatted at 2026-08-10 23:05:28 on dist-test-slave-6bbx"
server_key: "2e308fc720c9956ed87f5f0accff9d8a"
server_key_iv: "5b9d3531bf7cff3c237029f352f5d6b9"
server_key_version: "encryptionkey@0"
I20260810 23:05:28.418576 13473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:28.440912 13473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:28.441632 13473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:28.441794 13473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:28.442430 13473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:28.443475 13473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:28.443527 13473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:28.443572 13473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:28.443586 13473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:28.452646 13473 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:32945
I20260810 23:05:28.452695 13668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:32945 every 8 connection(s)
I20260810 23:05:28.453668 13473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:05:28.455186 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 13473
I20260810 23:05:28.455284 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:05:28.455557 15584 external_mini_cluster.cc:1428] Setting key 041aa5ed0ae3bf44f2557520e6d5b7a0
I20260810 23:05:28.464236 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.455733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:43797 (local address 127.15.56.62:33179)
0810 23:05:28.455902 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:28.455907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:28.457356 (+ 1449us) server_negotiation.cc:407] Connection header received
0810 23:05:28.458902 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:28.458905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:28.458964 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:28.459044 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:28.460135 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.460680 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.461920 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.462162 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.462271 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.463621 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:28.463624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:28.463626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:28.463641 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:28.463704 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:28.463716 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:28.463784 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:28.463967 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 23:05:28.464051 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:05:28.465286 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.456251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:33179 (local address 127.15.56.3:43797)
0810 23:05:28.457197 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:28.457262 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:28.458667 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:28.459243 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:28.459251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:28.459349 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:28.459981 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.459996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.460815 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.460818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.461786 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.461798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.462328 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.462332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.462513 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.463275 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:05:28.463277 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:28.463330 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:05:28.463420 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:28.463799 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:28.463811 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:28.463818 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:05:28.463966 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 23:05:28.464333 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":134,"threads_started":1}
I20260810 23:05:28.466799 13669 heartbeater.cc:347] Connected to a master server at 127.15.56.62:33179
I20260810 23:05:28.467067 13669 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:28.467657 13669 heartbeater.cc:511] Master 127.15.56.62:33179 requested a full tablet report, sending...
I20260810 23:05:28.468993 12966 ts_manager.cc:194] Registered new tserver with Master: a6e071bc9dee4f139d3d4e42aa758cb0 (127.15.56.3:32945)
I20260810 23:05:28.469738 12966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.3:43797
I20260810 23:05:28.469952 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:05:28.475813 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.472160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.15.56.62:33179)
0810 23:05:28.472300 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:28.472304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:28.472477 (+ 173us) server_negotiation.cc:407] Connection header received
0810 23:05:28.472570 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:28.472572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:28.472611 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:28.472673 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:28.472882 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.473390 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.474252 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.474495 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.474580 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.475362 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:28.475365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:28.475367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:28.475379 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:28.475400 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:28.475407 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:28.475509 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:28.475615 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 23:05:28.475646 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:05:28.478629 12966 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:05:28.481710 12966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:05:28.497215 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.490713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:34313 (local address 127.0.0.1:57112)
0810 23:05:28.491403 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:28.491417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:28.491512 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:28.493391 (+ 1879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:28.493394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:28.493411 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:28.493492 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.493498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.494597 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.494600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.495879 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.495892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.496816 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.496819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.496907 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.496928 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:05:28.496970 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 23:05:28.497025 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":76,"thread_start_us":107,"threads_started":1}
I20260810 23:05:28.497225 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.491542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.15.56.1:46131)
0810 23:05:28.491973 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:28.491986 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:28.492013 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:05:28.492103 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:28.492109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:28.492358 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:28.492517 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:28.492854 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.493996 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.495564 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.496427 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.496545 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.496780 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:28.496928 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 23:05:28.497046 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":188,"threads_started":1}
I20260810 23:05:28.497571 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.490945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:32945 (local address 127.0.0.1:37388)
0810 23:05:28.492170 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:28.492183 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:28.492268 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:28.493079 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:28.493082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:28.493099 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:28.493198 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.493205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.494483 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.494487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.495879 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.495892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.497194 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.497198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.497285 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.497308 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:05:28.497381 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 23:05:28.497451 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"mutex_wait_us":65,"thread_start_us":90,"threads_started":1}
I20260810 23:05:28.497215 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.490623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:46131 (local address 127.0.0.1:48642)
0810 23:05:28.491150 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:28.491203 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:28.491344 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:28.492518 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:28.492524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:28.492547 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:28.492702 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.492710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.494122 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.494126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.495429 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.495439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.496572 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.496576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.496701 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.496755 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 23:05:28.496886 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 23:05:28.497026 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":46,"thread_start_us":88,"threads_started":1}
I20260810 23:05:28.497816 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.492184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.15.56.2:34313)
0810 23:05:28.492966 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:28.492979 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:28.493004 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:05:28.493080 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:28.493086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:28.493280 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:28.493420 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:28.493587 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.494475 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.496035 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.496677 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.497289 (+ 612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.497499 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:28.497553 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 23:05:28.497656 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":99,"threads_started":1}
I20260810 23:05:28.498020 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.491975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.15.56.3:32945)
0810 23:05:28.492416 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:28.492429 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:28.492455 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:05:28.492533 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:28.492538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:28.492759 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:28.492930 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:28.493331 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.494369 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.496060 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.496828 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.496928 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.497112 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:28.497756 (+ 644us) server_negotiation.cc:299] Negotiation successful
0810 23:05:28.497865 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":105,"threads_started":1}
I20260810 23:05:28.500883 13570 tablet_service.cc:1467] Processing CreateTablet for tablet 8c9b4bba400c45ada91bd8b16a7beac4 (DEFAULT_TABLE table=test-table [id=f7a9c38471d84c30b7fe69b07bc8b38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:28.500941 13366 tablet_service.cc:1467] Processing CreateTablet for tablet 8c9b4bba400c45ada91bd8b16a7beac4 (DEFAULT_TABLE table=test-table [id=f7a9c38471d84c30b7fe69b07bc8b38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:28.500942 13163 tablet_service.cc:1467] Processing CreateTablet for tablet 8c9b4bba400c45ada91bd8b16a7beac4 (DEFAULT_TABLE table=test-table [id=f7a9c38471d84c30b7fe69b07bc8b38c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:28.501861 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c9b4bba400c45ada91bd8b16a7beac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:28.502005 13163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c9b4bba400c45ada91bd8b16a7beac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:28.502005 13366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c9b4bba400c45ada91bd8b16a7beac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:28.509130 13695 tablet_bootstrap.cc:492] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: Bootstrap starting.
I20260810 23:05:28.510658 13697 tablet_bootstrap.cc:492] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70: Bootstrap starting.
I20260810 23:05:28.511559 13695 tablet_bootstrap.cc:654] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:28.512380 13695 log.cc:824] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:28.512625 13697 tablet_bootstrap.cc:654] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:28.513321 13697 log.cc:824] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:28.514107 13696 tablet_bootstrap.cc:492] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88: Bootstrap starting.
I20260810 23:05:28.514546 13695 tablet_bootstrap.cc:492] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: No bootstrap required, opened a new log
I20260810 23:05:28.514756 13695 ts_tablet_manager.cc:1397] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:05:28.516312 13696 tablet_bootstrap.cc:654] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:28.517204 13696 log.cc:824] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:28.517549 13697 tablet_bootstrap.cc:492] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70: No bootstrap required, opened a new log
I20260810 23:05:28.517762 13697 ts_tablet_manager.cc:1397] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 23:05:28.518741 13695 raft_consensus.cc:348] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ead495ad7c446eea92d679a9f30dd88" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 46131 } } peers { permanent_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34313 } } peers { permanent_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 32945 } }
I20260810 23:05:28.518989 13695 raft_consensus.cc:374] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:28.519042 13695 raft_consensus.cc:729] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e071bc9dee4f139d3d4e42aa758cb0, State: Initialized, Role: FOLLOWER
I20260810 23:05:28.519550 13696 tablet_bootstrap.cc:492] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88: No bootstrap required, opened a new log
I20260810 23:05:28.519534 13695 consensus_queue.cc:260] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ead495ad7c446eea92d679a9f30dd88" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 46131 } } peers { permanent_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34313 } } peers { permanent_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 32945 } }
I20260810 23:05:28.519795 13696 ts_tablet_manager.cc:1397] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:05:28.520289 13669 heartbeater.cc:503] Master 127.15.56.62:33179 was elected leader, sending a full tablet report...
I20260810 23:05:28.520686 13695 ts_tablet_manager.cc:1428] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:05:28.522747 13697 raft_consensus.cc:348] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ead495ad7c446eea92d679a9f30dd88" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 46131 } } peers { permanent_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34313 } } peers { permanent_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 32945 } }
I20260810 23:05:28.523068 13697 raft_consensus.cc:374] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:28.523139 13697 raft_consensus.cc:729] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79adc951d814fc3aa03bb0c5b6c1b70, State: Initialized, Role: FOLLOWER
I20260810 23:05:28.523649 13697 consensus_queue.cc:260] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ead495ad7c446eea92d679a9f30dd88" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 46131 } } peers { permanent_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34313 } } peers { permanent_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 32945 } }
I20260810 23:05:28.524488 13468 heartbeater.cc:503] Master 127.15.56.62:33179 was elected leader, sending a full tablet report...
I20260810 23:05:28.524878 13697 ts_tablet_manager.cc:1428] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260810 23:05:28.525492 13696 raft_consensus.cc:348] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ead495ad7c446eea92d679a9f30dd88" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 46131 } } peers { permanent_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34313 } } peers { permanent_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 32945 } }
I20260810 23:05:28.525767 13696 raft_consensus.cc:374] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:28.525823 13696 raft_consensus.cc:729] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ead495ad7c446eea92d679a9f30dd88, State: Initialized, Role: FOLLOWER
I20260810 23:05:28.526268 13696 consensus_queue.cc:260] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ead495ad7c446eea92d679a9f30dd88" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 46131 } } peers { permanent_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34313 } } peers { permanent_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 32945 } }
I20260810 23:05:28.526983 13271 heartbeater.cc:503] Master 127.15.56.62:33179 was elected leader, sending a full tablet report...
I20260810 23:05:28.527369 13696 ts_tablet_manager.cc:1428] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260810 23:05:28.546470 13272 tablet.cc:2369] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:05:28.631886 13704 raft_consensus.cc:482] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:28.632136 13704 raft_consensus.cc:504] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ead495ad7c446eea92d679a9f30dd88" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 46131 } } peers { permanent_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34313 } } peers { permanent_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 32945 } }
I20260810 23:05:28.633374 13704 leader_election.cc:288] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f79adc951d814fc3aa03bb0c5b6c1b70 (127.15.56.2:34313), a6e071bc9dee4f139d3d4e42aa758cb0 (127.15.56.3:32945)
I20260810 23:05:28.637223 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.633756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:58401 (local address 127.15.56.2:34313)
0810 23:05:28.633890 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:28.633894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:28.634106 (+ 212us) server_negotiation.cc:407] Connection header received
0810 23:05:28.634264 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:28.634267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:28.634316 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:28.634382 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:28.634669 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.635165 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.636314 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.636698 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.636789 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.636854 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:28.637021 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 23:05:28.637092 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:05:28.637241 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.633763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:38807 (local address 127.15.56.3:32945)
0810 23:05:28.633890 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:28.633894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:28.634172 (+ 278us) server_negotiation.cc:407] Connection header received
0810 23:05:28.634332 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:28.634336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:28.634382 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:28.634444 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:28.634726 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.635216 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.636320 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.636715 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.636809 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.636870 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:28.637048 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 23:05:28.637120 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:05:28.637260 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.633662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:32945 (local address 127.15.56.1:38807)
0810 23:05:28.634069 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:28.634084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:28.634215 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:28.634471 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:28.634474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:28.634489 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:28.634612 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.634617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.635333 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.635337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.636189 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.636198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.636837 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.636841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.636942 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.636960 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 23:05:28.637023 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:05:28.637093 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":22,"thread_start_us":94,"threads_started":1}
I20260810 23:05:28.637272 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.633662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:34313 (local address 127.15.56.1:58401)
0810 23:05:28.634003 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:28.634016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:28.634122 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:28.634402 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:28.634405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:28.634421 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:28.634551 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.634558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.635272 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.635275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.636184 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:28.636195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.636809 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.636813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:28.636904 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.636930 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:05:28.636994 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 23:05:28.637065 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":4,"thread_start_us":100,"threads_started":1}
I20260810 23:05:28.637933 13600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9b4bba400c45ada91bd8b16a7beac4" candidate_uuid: "9ead495ad7c446eea92d679a9f30dd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" is_pre_election: true
I20260810 23:05:28.638247 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9b4bba400c45ada91bd8b16a7beac4" candidate_uuid: "9ead495ad7c446eea92d679a9f30dd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" is_pre_election: true
I20260810 23:05:28.638293 13600 raft_consensus.cc:2454] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ead495ad7c446eea92d679a9f30dd88 in term 0.
I20260810 23:05:28.638538 13402 raft_consensus.cc:2454] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ead495ad7c446eea92d679a9f30dd88 in term 0.
I20260810 23:05:28.638808 13078 leader_election.cc:302] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ead495ad7c446eea92d679a9f30dd88, a6e071bc9dee4f139d3d4e42aa758cb0; no voters:
I20260810 23:05:28.639081 13704 raft_consensus.cc:2781] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:05:28.639168 13704 raft_consensus.cc:482] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:05:28.639212 13704 raft_consensus.cc:3037] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:28.640340 13704 raft_consensus.cc:504] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ead495ad7c446eea92d679a9f30dd88" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 46131 } } peers { permanent_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34313 } } peers { permanent_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 32945 } }
I20260810 23:05:28.640652 13704 leader_election.cc:288] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [CANDIDATE]: Term 1 election: Requested vote from peers f79adc951d814fc3aa03bb0c5b6c1b70 (127.15.56.2:34313), a6e071bc9dee4f139d3d4e42aa758cb0 (127.15.56.3:32945)
I20260810 23:05:28.641017 13600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9b4bba400c45ada91bd8b16a7beac4" candidate_uuid: "9ead495ad7c446eea92d679a9f30dd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0"
I20260810 23:05:28.641037 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9b4bba400c45ada91bd8b16a7beac4" candidate_uuid: "9ead495ad7c446eea92d679a9f30dd88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70"
I20260810 23:05:28.641135 13600 raft_consensus.cc:3037] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:28.641153 13402 raft_consensus.cc:3037] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:28.642185 13600 raft_consensus.cc:2454] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ead495ad7c446eea92d679a9f30dd88 in term 1.
I20260810 23:05:28.642257 13402 raft_consensus.cc:2454] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ead495ad7c446eea92d679a9f30dd88 in term 1.
I20260810 23:05:28.642482 13078 leader_election.cc:302] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ead495ad7c446eea92d679a9f30dd88, a6e071bc9dee4f139d3d4e42aa758cb0; no voters:
I20260810 23:05:28.642661 13704 raft_consensus.cc:2781] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:28.642900 13704 raft_consensus.cc:686] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [term 1 LEADER]: Becoming Leader. State: Replica: 9ead495ad7c446eea92d679a9f30dd88, State: Running, Role: LEADER
I20260810 23:05:28.643217 13704 consensus_queue.cc:237] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ead495ad7c446eea92d679a9f30dd88" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 46131 } } peers { permanent_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34313 } } peers { permanent_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 32945 } }
I20260810 23:05:28.646526 12966 catalog_manager.cc:5495] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ead495ad7c446eea92d679a9f30dd88 (127.15.56.1). New cstate: current_term: 1 leader_uuid: "9ead495ad7c446eea92d679a9f30dd88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ead495ad7c446eea92d679a9f30dd88" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 46131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 32945 } health_report { overall_health: UNKNOWN } } }
I20260810 23:05:28.691515 13686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.688008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.15.56.1:46131)
0810 23:05:28.688152 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:28.688161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:28.688178 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:05:28.688257 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:28.688260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:28.688319 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:28.688413 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:05:28.688651 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.689187 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.690014 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:28.690248 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:28.690332 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:28.690572 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:05:28.691096 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:05:28.691241 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:28.691321 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 23:05:28.691364 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:05:28.698025 13600 raft_consensus.cc:1261] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0 [term 1 FOLLOWER]: Refusing update from remote peer 9ead495ad7c446eea92d679a9f30dd88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:28.698025 13402 raft_consensus.cc:1261] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70 [term 1 FOLLOWER]: Refusing update from remote peer 9ead495ad7c446eea92d679a9f30dd88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:28.698777 13704 consensus_queue.cc:1035] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e071bc9dee4f139d3d4e42aa758cb0" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:28.699007 13709 consensus_queue.cc:1035] T 8c9b4bba400c45ada91bd8b16a7beac4 P 9ead495ad7c446eea92d679a9f30dd88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f79adc951d814fc3aa03bb0c5b6c1b70" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:05:28.704741 13670 tablet.cc:2369] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:05:28.706629 13717 mvcc.cc:205] Tried to move back new op lower bound from 7317107215139078144 to 7317107214925549568. Current Snapshot: MvccSnapshot[applied={T|T < 7317107215139078144}]
I20260810 23:05:28.709756 13716 mvcc.cc:205] Tried to move back new op lower bound from 7317107215139078144 to 7317107214925549568. Current Snapshot: MvccSnapshot[applied={T|T < 7317107215139078144}]
I20260810 23:05:28.715528 12967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:28.715772 12967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:28.718698 12967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef0f9fa18df47a2931289cd0cfc55d6: Sending DeleteTablet for 3 replicas of tablet 8c9b4bba400c45ada91bd8b16a7beac4
I20260810 23:05:28.719570 13366 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c9b4bba400c45ada91bd8b16a7beac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:28 UTC) from {username='slave'} at 127.0.0.1:57112
I20260810 23:05:28.719887 13570 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c9b4bba400c45ada91bd8b16a7beac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:28 UTC) from {username='slave'} at 127.0.0.1:37388
I20260810 23:05:28.720367 13728 tablet_replica.cc:330] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: stopping tablet replica
I20260810 23:05:28.719570 13163 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c9b4bba400c45ada91bd8b16a7beac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:28 UTC) from {username='slave'} at 127.0.0.1:48642
I20260810 23:05:28.720723 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 13057
I20260810 23:05:28.745015 13727 tablet_replica.cc:330] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70: stopping tablet replica
I20260810 23:05:28.745479 13727 raft_consensus.cc:2229] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:28.745865 13727 raft_consensus.cc:2258] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:28.751776 13727 ts_tablet_manager.cc:1905] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:28.754632 13727 ts_tablet_manager.cc:1918] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:28.754765 13727 log.cc:1196] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/8c9b4bba400c45ada91bd8b16a7beac4
I20260810 23:05:28.755029 13727 ts_tablet_manager.cc:1939] T 8c9b4bba400c45ada91bd8b16a7beac4 P f79adc951d814fc3aa03bb0c5b6c1b70: Deleting consensus metadata
I20260810 23:05:28.757079 12946 catalog_manager.cc:4841] TS f79adc951d814fc3aa03bb0c5b6c1b70 (127.15.56.2:34313): tablet 8c9b4bba400c45ada91bd8b16a7beac4 (table test-table [id=f7a9c38471d84c30b7fe69b07bc8b38c]) successfully deleted
I20260810 23:05:28.720800 13728 raft_consensus.cc:2229] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:28.766363 13728 raft_consensus.cc:2258] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:28.769266 13728 ts_tablet_manager.cc:1905] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:28.771814 13728 ts_tablet_manager.cc:1918] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:28.771920 13728 log.cc:1196] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/8c9b4bba400c45ada91bd8b16a7beac4
I20260810 23:05:28.772195 13728 ts_tablet_manager.cc:1939] T 8c9b4bba400c45ada91bd8b16a7beac4 P a6e071bc9dee4f139d3d4e42aa758cb0: Deleting consensus metadata
I20260810 23:05:28.773450 12943 catalog_manager.cc:4841] TS a6e071bc9dee4f139d3d4e42aa758cb0 (127.15.56.3:32945): tablet 8c9b4bba400c45ada91bd8b16a7beac4 (table test-table [id=f7a9c38471d84c30b7fe69b07bc8b38c]) successfully deleted
W20260810 23:05:28.776335 12950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:46131 (error 108)
I20260810 23:05:28.777211 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 13274
I20260810 23:05:28.777590 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:28.777218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:46131 (local address 127.0.0.1:48664)
0810 23:05:28.777407 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:28.777494 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:46131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260810 23:05:28.797155 12950 catalog_manager.cc:4584] TS 9ead495ad7c446eea92d679a9f30dd88 (127.15.56.1:46131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c9b4bba400c45ada91bd8b16a7beac4: Network error: Client connection negotiation failed: client connection to 127.15.56.1:46131: connect: Connection refused (error 111)
I20260810 23:05:28.817198 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 13473
I20260810 23:05:28.832036 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 12921
2026-08-10T23:05:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1267 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:28 dist-test-slave-6bbx krb5kdc[13739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:28 dist-test-slave-6bbx krb5kdc[13739](info): set up 2 sockets
Aug 10 23:05:28 dist-test-slave-6bbx krb5kdc[13739](info): commencing operation
krb5kdc: starting...
W20260810 23:05:30.008847 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:30 dist-test-slave-6bbx krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:05:30.159235 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:40673
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:40673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:30.277082 13758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:30.277411 13758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:30.277455 13758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:30.281606 13758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:30.281688 13758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:30.281708 13758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:30.281723 13758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:30.281737 13758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:30.287578 13758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:40673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:40673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:30.288859 13758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:30.290004 13758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:05:30.299124 13758 server_base.cc:1034] running on GCE node
W20260810 23:05:30.299050 13770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:30.299079 13766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:30.299041 13765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:30.299957 13758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:30.301136 13758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:30.302311 13758 hybrid_clock.cc:648] HybridClock initialized: now 1786403130302293 us; error 27 us; skew 500 ppm
Aug 10 23:05:30 dist-test-slave-6bbx krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403130, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:30.305171 13758 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:05:30.306095 13758 webserver.cc:460] Webserver started at http://127.15.56.62:36297/ using document root <none> and password file <none>
I20260810 23:05:30.306679 13758 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:30.306742 13758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:30.306937 13758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:30.309060 13758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "65fbac9f524945baad1ac94fcae5f7c7"
format_stamp: "Formatted at 2026-08-10 23:05:30 on dist-test-slave-6bbx"
server_key: "c1e9ee5213443049f040483206c5631f"
server_key_iv: "518b814c2b99c39ee2a75660f1ab6fc5"
server_key_version: "encryptionkey@0"
I20260810 23:05:30.309582 13758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "65fbac9f524945baad1ac94fcae5f7c7"
format_stamp: "Formatted at 2026-08-10 23:05:30 on dist-test-slave-6bbx"
server_key: "c1e9ee5213443049f040483206c5631f"
server_key_iv: "518b814c2b99c39ee2a75660f1ab6fc5"
server_key_version: "encryptionkey@0"
I20260810 23:05:30.313272 13758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:05:30.315758 13779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:30.317039 13758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:05:30.317149 13758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "65fbac9f524945baad1ac94fcae5f7c7"
format_stamp: "Formatted at 2026-08-10 23:05:30 on dist-test-slave-6bbx"
server_key: "c1e9ee5213443049f040483206c5631f"
server_key_iv: "518b814c2b99c39ee2a75660f1ab6fc5"
server_key_version: "encryptionkey@0"
I20260810 23:05:30.317260 13758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:30.336069 13758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:30.339175 13758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:30.339372 13758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:30.346483 13758 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:40673
I20260810 23:05:30.346505 13865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:40673 every 8 connection(s)
I20260810 23:05:30.347506 13758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:30.350441 13866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:30.356390 13866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7: Bootstrap starting.
I20260810 23:05:30.357172 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 13758
I20260810 23:05:30.357301 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:30.357548 15584 external_mini_cluster.cc:1428] Setting key ebc3c478396e1a63da6a62182cef4935
I20260810 23:05:30.359336 13866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:30.360277 13866 log.cc:824] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:30.362351 13866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7: No bootstrap required, opened a new log
Aug 10 23:05:30 dist-test-slave-6bbx krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:05:30.366212 13866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fbac9f524945baad1ac94fcae5f7c7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40673 } }
I20260810 23:05:30.366496 13866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:30.366559 13866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fbac9f524945baad1ac94fcae5f7c7, State: Initialized, Role: FOLLOWER
I20260810 23:05:30.367024 13866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fbac9f524945baad1ac94fcae5f7c7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40673 } }
I20260810 23:05:30.367173 13866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:30.367247 13866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:30.367358 13866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:30.368444 13866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fbac9f524945baad1ac94fcae5f7c7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40673 } }
I20260810 23:05:30.368820 13866 leader_election.cc:302] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65fbac9f524945baad1ac94fcae5f7c7; no voters:
I20260810 23:05:30.369139 13866 leader_election.cc:288] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:30.369233 13874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:30.369405 13874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [term 1 LEADER]: Becoming Leader. State: Replica: 65fbac9f524945baad1ac94fcae5f7c7, State: Running, Role: LEADER
I20260810 23:05:30.369722 13874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fbac9f524945baad1ac94fcae5f7c7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40673 } }
I20260810 23:05:30.370532 13866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:30.372058 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:30.358754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.15.56.62:40673)
0810 23:05:30.359209 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:30.359226 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:30.359265 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:05:30.360058 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:30.360079 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:30.360470 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:30.360810 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:30.361636 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:30.362723 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:30.363595 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:30.364123 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:30.364275 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:30.366801 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:30.366808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:30.366828 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:30.366865 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:30.368637 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:30.369127 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:30.369134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:30.369140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:30.369290 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:30.369683 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:30.369686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:30.369688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:30.370152 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:30.370294 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:30.370727 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 23:05:30.371057 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":134,"threads_started":1}
I20260810 23:05:30.372884 13876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65fbac9f524945baad1ac94fcae5f7c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fbac9f524945baad1ac94fcae5f7c7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40673 } } }
I20260810 23:05:30.373121 13876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:30.373304 13875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65fbac9f524945baad1ac94fcae5f7c7. Latest consensus state: current_term: 1 leader_uuid: "65fbac9f524945baad1ac94fcae5f7c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fbac9f524945baad1ac94fcae5f7c7" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40673 } } }
I20260810 23:05:30.373382 13875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7 [sys.catalog]: This master's current role is: LEADER
W20260810 23:05:30.375416 13893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:05:30.375492 13893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:30.375576 13894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:30.376520 13894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:30.382790 13894 catalog_manager.cc:1347] Generated new cluster ID: 1a9b942d117f4ea4be7d7baf38e094fc
I20260810 23:05:30.382884 13894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:30.395290 13894 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:30.396363 13894 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:30.401029 13894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7: Generated new TSK 0
I20260810 23:05:30.401889 13894 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.56.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
Entry for principal kudu/127.15.56.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab.
I20260810 23:05:30.434370 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40673
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:30.554461 13902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:30.554773 13902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:30.554817 13902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:30.558930 13902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:30.558993 13902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:30.559056 13902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:05:30.564478 13902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.1.keytab
--principal=kudu/127.15.56.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:30.565529 13902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:30.566552 13902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:30.576838 13919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:30.576848 13925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:30.577013 13902 server_base.cc:1034] running on GCE node
W20260810 23:05:30.576920 13910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:30.577883 13902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:30.578559 13902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:30.579744 13902 hybrid_clock.cc:648] HybridClock initialized: now 1786403130579717 us; error 35 us; skew 500 ppm
Aug 10 23:05:30 dist-test-slave-6bbx krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403130, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:30.582532 13902 init.cc:377] Logged in from keytab as kudu/127.15.56.1@KRBTEST.COM (short username kudu)
I20260810 23:05:30.583439 13902 webserver.cc:460] Webserver started at http://127.15.56.1:41111/ using document root <none> and password file <none>
I20260810 23:05:30.584043 13902 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:30.584115 13902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:30.584316 13902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:30.586426 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "6173bac0950347b69ebd912e51b7783d"
format_stamp: "Formatted at 2026-08-10 23:05:30 on dist-test-slave-6bbx"
server_key: "0d6a967f59460e3d702979b96c3240bf"
server_key_iv: "7f477ed56b402f2557b008e524490677"
server_key_version: "encryptionkey@0"
I20260810 23:05:30.586980 13902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "6173bac0950347b69ebd912e51b7783d"
format_stamp: "Formatted at 2026-08-10 23:05:30 on dist-test-slave-6bbx"
server_key: "0d6a967f59460e3d702979b96c3240bf"
server_key_iv: "7f477ed56b402f2557b008e524490677"
server_key_version: "encryptionkey@0"
I20260810 23:05:30.590806 13902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:05:30.593343 13936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:30.594583 13902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:05:30.594715 13902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "6173bac0950347b69ebd912e51b7783d"
format_stamp: "Formatted at 2026-08-10 23:05:30 on dist-test-slave-6bbx"
server_key: "0d6a967f59460e3d702979b96c3240bf"
server_key_iv: "7f477ed56b402f2557b008e524490677"
server_key_version: "encryptionkey@0"
I20260810 23:05:30.594820 13902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:30.614437 13902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:30.617592 13902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:30.617791 13902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:30.618424 13902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:30.619334 13902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:30.619385 13902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:30.619441 13902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:30.619469 13902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:30.628281 13902 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:38923
I20260810 23:05:30.628369 14105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:38923 every 8 connection(s)
I20260810 23:05:30.629262 13902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:30.631821 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 13902
I20260810 23:05:30.631968 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:05:30.632339 15584 external_mini_cluster.cc:1428] Setting key 2740bc55736c24175a03539346186a95
Aug 10 23:05:30 dist-test-slave-6bbx krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403130, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.1@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.2@KRBTEST.COM; defaulting to no policy
I20260810 23:05:30.644809 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:30.631217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:48071 (local address 127.15.56.62:40673)
0810 23:05:30.631365 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:30.631369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:30.632682 (+ 1313us) server_negotiation.cc:407] Connection header received
0810 23:05:30.633767 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:30.633770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:30.633828 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:30.633922 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:30.635658 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:30.636452 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:30.637658 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:30.637972 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:30.638082 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:30.640992 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:30.640998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:30.641001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:30.641036 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:30.642661 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:30.643214 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:30.643219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:30.643221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:30.643322 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:30.643637 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:30.643640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:30.643642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:30.643937 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:30.644045 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:30.644538 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 23:05:30.644659 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:05:30.645802 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:30.631590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40673 (local address 127.15.56.1:48071)
0810 23:05:30.632489 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:30.632556 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:30.633519 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:30.634194 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:30.634205 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:30.634754 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:30.635473 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:30.635495 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:30.636588 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:30.636591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:30.637525 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:30.637535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:30.638151 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:30.638154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:30.638309 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:30.638933 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:30.638950 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:30.640783 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:30.642803 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:30.642809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:30.642827 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:30.643103 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:30.643414 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:30.643416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:30.643417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:30.643556 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:30.644038 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:30.644043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:30.644307 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 23:05:30.644485 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 23:05:30.644824 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.2@KRBTEST.COM" created.
I20260810 23:05:30.647377 14106 heartbeater.cc:347] Connected to a master server at 127.15.56.62:40673
I20260810 23:05:30.647650 14106 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:30.648278 14106 heartbeater.cc:511] Master 127.15.56.62:40673 requested a full tablet report, sending...
I20260810 23:05:30.649921 13808 ts_manager.cc:194] Registered new tserver with Master: 6173bac0950347b69ebd912e51b7783d (127.15.56.1:38923)
I20260810 23:05:30.651427 13808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.1@KRBTEST.COM'} at 127.15.56.1:48071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
Entry for principal kudu/127.15.56.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab.
I20260810 23:05:30.663944 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40673
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:30.784178 14112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:30.784476 14112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:30.784521 14112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:30.788690 14112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:30.788766 14112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:30.788827 14112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:05:30.794364 14112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.2.keytab
--principal=kudu/127.15.56.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:30.795460 14112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:30.796613 14112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:30.806977 14125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:30.806963 14122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:30.806963 14128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:30.807454 14112 server_base.cc:1034] running on GCE node
I20260810 23:05:30.807979 14112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:30.808710 14112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:30.809900 14112 hybrid_clock.cc:648] HybridClock initialized: now 1786403130809884 us; error 25 us; skew 500 ppm
Aug 10 23:05:30 dist-test-slave-6bbx krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403130, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:30.812886 14112 init.cc:377] Logged in from keytab as kudu/127.15.56.2@KRBTEST.COM (short username kudu)
I20260810 23:05:30.813867 14112 webserver.cc:460] Webserver started at http://127.15.56.2:37745/ using document root <none> and password file <none>
I20260810 23:05:30.814452 14112 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:30.814527 14112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:30.814752 14112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:30.817019 14112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "fe487574e25b48c29a09551b86946296"
format_stamp: "Formatted at 2026-08-10 23:05:30 on dist-test-slave-6bbx"
server_key: "392b2a95446bb65aa894e69d571c8608"
server_key_iv: "0299d26502fcffa484258b9c1851badd"
server_key_version: "encryptionkey@0"
I20260810 23:05:30.817598 14112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "fe487574e25b48c29a09551b86946296"
format_stamp: "Formatted at 2026-08-10 23:05:30 on dist-test-slave-6bbx"
server_key: "392b2a95446bb65aa894e69d571c8608"
server_key_iv: "0299d26502fcffa484258b9c1851badd"
server_key_version: "encryptionkey@0"
I20260810 23:05:30.821718 14112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:05:30.824564 14139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:30.825875 14112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:30.826002 14112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "fe487574e25b48c29a09551b86946296"
format_stamp: "Formatted at 2026-08-10 23:05:30 on dist-test-slave-6bbx"
server_key: "392b2a95446bb65aa894e69d571c8608"
server_key_iv: "0299d26502fcffa484258b9c1851badd"
server_key_version: "encryptionkey@0"
I20260810 23:05:30.826133 14112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:30.844265 14112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:30.847359 14112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:30.847550 14112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:30.848233 14112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:30.849102 14112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:30.849151 14112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:30.849218 14112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:30.849244 14112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:30.858114 14112 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:35441
I20260810 23:05:30.858137 14315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:35441 every 8 connection(s)
I20260810 23:05:30.859061 14112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:30.862130 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 14112
I20260810 23:05:30.862273 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:05:30.862555 15584 external_mini_cluster.cc:1428] Setting key 130100bf6e419c7082beccb77d36ac22
Aug 10 23:05:30 dist-test-slave-6bbx krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403130, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.2@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.3@KRBTEST.COM; defaulting to no policy
I20260810 23:05:30.875620 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:30.860979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:52161 (local address 127.15.56.62:40673)
0810 23:05:30.861173 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:30.861178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:30.862269 (+ 1091us) server_negotiation.cc:407] Connection header received
0810 23:05:30.863376 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:30.863379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:30.863423 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:30.863494 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:30.864932 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:30.865572 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:30.867210 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:30.867433 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:30.867529 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:30.870914 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:30.870920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:30.870923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:30.870989 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:30.873044 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:30.873838 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:30.873843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:30.873844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:30.873953 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:30.874342 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:30.874346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:30.874348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:30.874622 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:30.874738 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:30.875353 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 23:05:30.875479 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":64}
I20260810 23:05:30.876669 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:30.861359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40673 (local address 127.15.56.2:52161)
0810 23:05:30.862072 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:30.862159 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:30.863170 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:30.863707 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:30.863717 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:30.864258 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:30.864777 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:30.864793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:30.865743 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:30.865746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:30.867034 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:30.867047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:30.867653 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:30.867658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:30.867935 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:30.868649 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:30.868672 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:30.870665 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:30.873340 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:30.873348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:30.873366 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:30.873698 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:30.874054 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:30.874056 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:30.874057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:30.874223 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:30.874750 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:30.874758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:30.875059 (+ 301us) client_negotiation.cc:769] Sending connection context
0810 23:05:30.875288 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 23:05:30.875617 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.3@KRBTEST.COM" created.
I20260810 23:05:30.878005 14316 heartbeater.cc:347] Connected to a master server at 127.15.56.62:40673
I20260810 23:05:30.878306 14316 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:30.878850 14316 heartbeater.cc:511] Master 127.15.56.62:40673 requested a full tablet report, sending...
I20260810 23:05:30.880098 13808 ts_manager.cc:194] Registered new tserver with Master: fe487574e25b48c29a09551b86946296 (127.15.56.2:35441)
I20260810 23:05:30.880728 13808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.2@KRBTEST.COM'} at 127.15.56.2:52161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
Entry for principal kudu/127.15.56.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab.
I20260810 23:05:30.893975 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40673
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:31.011782 14322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:31.012084 14322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:31.012154 14322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:31.016409 14322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:31.016496 14322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:31.016580 14322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:05:31.022502 14322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.3.keytab
--principal=kudu/127.15.56.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:40673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:31.023783 14322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:31.024910 14322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:05:31.035238 14322 server_base.cc:1034] running on GCE node
W20260810 23:05:31.035193 14332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:31.035183 14327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:31.035183 14334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:31.035921 14322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:31.036562 14322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:31.037724 14322 hybrid_clock.cc:648] HybridClock initialized: now 1786403131037714 us; error 37 us; skew 500 ppm
Aug 10 23:05:31 dist-test-slave-6bbx krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403131, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:31.040542 14322 init.cc:377] Logged in from keytab as kudu/127.15.56.3@KRBTEST.COM (short username kudu)
I20260810 23:05:31.041438 14322 webserver.cc:460] Webserver started at http://127.15.56.3:38647/ using document root <none> and password file <none>
I20260810 23:05:31.042038 14322 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:31.042115 14322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:31.042335 14322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:31.044533 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "3a856b77389a4578b24d490231334fbe"
format_stamp: "Formatted at 2026-08-10 23:05:31 on dist-test-slave-6bbx"
server_key: "8057436e2491deb32eaf2cb1187cc12f"
server_key_iv: "8ba7bfbbc13500733edf34241678e516"
server_key_version: "encryptionkey@0"
I20260810 23:05:31.045110 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "3a856b77389a4578b24d490231334fbe"
format_stamp: "Formatted at 2026-08-10 23:05:31 on dist-test-slave-6bbx"
server_key: "8057436e2491deb32eaf2cb1187cc12f"
server_key_iv: "8ba7bfbbc13500733edf34241678e516"
server_key_version: "encryptionkey@0"
I20260810 23:05:31.048988 14322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:31.051761 14346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:31.053153 14322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:31.053267 14322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "3a856b77389a4578b24d490231334fbe"
format_stamp: "Formatted at 2026-08-10 23:05:31 on dist-test-slave-6bbx"
server_key: "8057436e2491deb32eaf2cb1187cc12f"
server_key_iv: "8ba7bfbbc13500733edf34241678e516"
server_key_version: "encryptionkey@0"
I20260810 23:05:31.053392 14322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:31.068542 14322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:31.072463 14322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:31.072677 14322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:31.073357 14322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:31.074340 14322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:31.074404 14322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:31.074463 14322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:31.074492 14322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:31.084755 14322 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:38419
I20260810 23:05:31.084790 14530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:38419 every 8 connection(s)
I20260810 23:05:31.085721 14322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:05:31.091789 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 14322
I20260810 23:05:31.091943 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:05:31.092281 15584 external_mini_cluster.cc:1428] Setting key aa7d69440ebbf4990485069b3256eb05
Aug 10 23:05:31 dist-test-slave-6bbx krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403131, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.3@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:05:31.100455 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.087620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:40231 (local address 127.15.56.62:40673)
0810 23:05:31.087810 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:31.087815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:31.088945 (+ 1130us) server_negotiation.cc:407] Connection header received
0810 23:05:31.089942 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:31.089945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:31.089996 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:31.090079 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:31.091344 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.092166 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.093652 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.093953 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.094061 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.096711 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:31.096715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:31.096717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:31.096744 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:31.098090 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:31.098677 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:31.098681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:31.098684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:31.098768 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:31.099112 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:31.099115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:31.099117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:31.099319 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:31.099410 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:31.100125 (+ 715us) server_negotiation.cc:299] Negotiation successful
0810 23:05:31.100273 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 23:05:31.101600 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.088084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:40673 (local address 127.15.56.3:40231)
0810 23:05:31.088797 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:31.088841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:31.089649 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:31.090276 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:31.090284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:31.090662 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:31.091177 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.091193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.092336 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.092339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.093528 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.093543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.094096 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.094099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.094266 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.094898 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:31.094916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:31.096513 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:31.098208 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:31.098214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:31.098230 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:31.098537 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:31.098857 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:31.098860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:31.098862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:31.099021 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:31.099422 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:31.099429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:31.099773 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 23:05:31.100058 (+ 285us) client_negotiation.cc:240] Negotiation successful
0810 23:05:31.100473 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":144,"threads_started":1}
I20260810 23:05:31.103029 14531 heartbeater.cc:347] Connected to a master server at 127.15.56.62:40673
I20260810 23:05:31.103317 14531 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:31.103964 14531 heartbeater.cc:511] Master 127.15.56.62:40673 requested a full tablet report, sending...
I20260810 23:05:31.105237 13808 ts_manager.cc:194] Registered new tserver with Master: 3a856b77389a4578b24d490231334fbe (127.15.56.3:38419)
I20260810 23:05:31.105930 13808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.56.3@KRBTEST.COM'} at 127.15.56.3:40231
I20260810 23:05:31.106632 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:05:31.116238 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.108752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.15.56.62:40673)
0810 23:05:31.108874 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:31.108878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:31.109064 (+ 186us) server_negotiation.cc:407] Connection header received
0810 23:05:31.109169 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:31.109171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:31.109211 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:31.109282 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:31.110007 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.110508 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.111403 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.111623 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.111788 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.113175 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:31.113179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:31.113181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:31.113206 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:31.114591 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:31.115025 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:31.115028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:31.115030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:31.115117 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:31.115435 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:31.115436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:31.115438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:31.115658 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:31.115805 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:31.116007 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 23:05:31.116113 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:05:31.119558 13807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:05:31.122746 13807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:05:31.137938 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.131744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:35441 (local address 127.0.0.1:35536)
0810 23:05:31.132250 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:31.132304 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:31.132438 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:31.133306 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:31.133312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:31.133338 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:31.133480 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.133487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.134893 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.134896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.136169 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.136181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.137344 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.137348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.137462 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.137506 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 23:05:31.137629 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 23:05:31.137767 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":95,"threads_started":1}
I20260810 23:05:31.137949 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.131873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.15.56.2:35441)
0810 23:05:31.132567 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:31.132580 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:31.132608 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:05:31.132692 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:31.132697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:31.132939 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:31.133092 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:31.133614 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.134735 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.136315 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.137210 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.137339 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.137505 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:31.137683 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 23:05:31.137788 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":121,"threads_started":1}
I20260810 23:05:31.138469 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.131898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:38923 (local address 127.0.0.1:53182)
0810 23:05:31.132828 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:31.132843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:31.132937 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:31.133838 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:31.133841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:31.133868 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:31.134024 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.134032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.135413 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.135417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.136577 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.136587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.137903 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.137907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.138005 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.138189 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 23:05:31.138256 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 23:05:31.138327 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":27,"spinlock_wait_cycles":1792,"thread_start_us":113,"threads_started":1}
I20260810 23:05:31.138633 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.132811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.15.56.3:38419)
0810 23:05:31.133234 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:31.133244 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:31.133263 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:05:31.133331 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:31.133336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:31.133526 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:31.133679 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:31.134938 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.135864 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.137105 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.137859 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.137957 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.138098 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:31.138406 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 23:05:31.138497 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":91,"threads_started":1}
I20260810 23:05:31.138777 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.132422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.15.56.1:38923)
0810 23:05:31.133211 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:31.133226 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:31.133253 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:05:31.133331 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:31.133336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:31.133525 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:31.133679 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:31.134164 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.135242 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.136768 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.137594 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.137697 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.137835 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:31.138550 (+ 715us) server_negotiation.cc:299] Negotiation successful
0810 23:05:31.138642 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":107,"threads_started":1}
I20260810 23:05:31.138737 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.132503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:38419 (local address 127.0.0.1:35992)
0810 23:05:31.132911 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:31.132923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:31.133014 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:31.133838 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:31.133841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:31.133874 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:31.134037 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.134043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.136019 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.136023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.136983 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.136993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.138162 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.138165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.138262 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.138298 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 23:05:31.138368 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:05:31.138439 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":46,"spinlock_wait_cycles":5248,"thread_start_us":125,"threads_started":1}
I20260810 23:05:31.141331 14210 tablet_service.cc:1467] Processing CreateTablet for tablet 998e5cb3d81a46db8171c74b82ba615e (DEFAULT_TABLE table=test-table [id=a994e1fb050d4cae9685c2cb236700eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:31.141706 14005 tablet_service.cc:1467] Processing CreateTablet for tablet 998e5cb3d81a46db8171c74b82ba615e (DEFAULT_TABLE table=test-table [id=a994e1fb050d4cae9685c2cb236700eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:31.142073 14426 tablet_service.cc:1467] Processing CreateTablet for tablet 998e5cb3d81a46db8171c74b82ba615e (DEFAULT_TABLE table=test-table [id=a994e1fb050d4cae9685c2cb236700eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:31.142536 14210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998e5cb3d81a46db8171c74b82ba615e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:31.142908 14005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998e5cb3d81a46db8171c74b82ba615e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:31.143159 14426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998e5cb3d81a46db8171c74b82ba615e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:31.149866 14558 tablet_bootstrap.cc:492] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296: Bootstrap starting.
I20260810 23:05:31.151597 14562 tablet_bootstrap.cc:492] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: Bootstrap starting.
I20260810 23:05:31.152029 14559 tablet_bootstrap.cc:492] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d: Bootstrap starting.
I20260810 23:05:31.153590 14558 tablet_bootstrap.cc:654] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:31.154124 14559 tablet_bootstrap.cc:654] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:31.154114 14562 tablet_bootstrap.cc:654] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:31.154455 14558 log.cc:824] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:31.155031 14559 log.cc:824] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:31.155350 14562 log.cc:824] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:31.156924 14558 tablet_bootstrap.cc:492] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296: No bootstrap required, opened a new log
I20260810 23:05:31.157127 14559 tablet_bootstrap.cc:492] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d: No bootstrap required, opened a new log
I20260810 23:05:31.157202 14558 ts_tablet_manager.cc:1397] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 23:05:31.157351 14559 ts_tablet_manager.cc:1397] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:05:31.157539 14562 tablet_bootstrap.cc:492] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: No bootstrap required, opened a new log
I20260810 23:05:31.157709 14562 ts_tablet_manager.cc:1397] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20260810 23:05:31.161178 14559 raft_consensus.cc:348] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe487574e25b48c29a09551b86946296" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35441 } } peers { permanent_uuid: "6173bac0950347b69ebd912e51b7783d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 38923 } } peers { permanent_uuid: "3a856b77389a4578b24d490231334fbe" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 38419 } }
I20260810 23:05:31.161443 14559 raft_consensus.cc:374] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:31.161500 14559 raft_consensus.cc:729] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6173bac0950347b69ebd912e51b7783d, State: Initialized, Role: FOLLOWER
I20260810 23:05:31.161367 14558 raft_consensus.cc:348] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe487574e25b48c29a09551b86946296" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35441 } } peers { permanent_uuid: "6173bac0950347b69ebd912e51b7783d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 38923 } } peers { permanent_uuid: "3a856b77389a4578b24d490231334fbe" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 38419 } }
I20260810 23:05:31.161458 14562 raft_consensus.cc:348] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe487574e25b48c29a09551b86946296" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35441 } } peers { permanent_uuid: "6173bac0950347b69ebd912e51b7783d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 38923 } } peers { permanent_uuid: "3a856b77389a4578b24d490231334fbe" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 38419 } }
I20260810 23:05:31.161674 14562 raft_consensus.cc:374] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:31.161674 14558 raft_consensus.cc:374] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:31.161723 14558 raft_consensus.cc:729] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe487574e25b48c29a09551b86946296, State: Initialized, Role: FOLLOWER
I20260810 23:05:31.161723 14562 raft_consensus.cc:729] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a856b77389a4578b24d490231334fbe, State: Initialized, Role: FOLLOWER
I20260810 23:05:31.162060 14559 consensus_queue.cc:260] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe487574e25b48c29a09551b86946296" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35441 } } peers { permanent_uuid: "6173bac0950347b69ebd912e51b7783d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 38923 } } peers { permanent_uuid: "3a856b77389a4578b24d490231334fbe" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 38419 } }
I20260810 23:05:31.162204 14558 consensus_queue.cc:260] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe487574e25b48c29a09551b86946296" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35441 } } peers { permanent_uuid: "6173bac0950347b69ebd912e51b7783d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 38923 } } peers { permanent_uuid: "3a856b77389a4578b24d490231334fbe" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 38419 } }
I20260810 23:05:31.162204 14562 consensus_queue.cc:260] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe487574e25b48c29a09551b86946296" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35441 } } peers { permanent_uuid: "6173bac0950347b69ebd912e51b7783d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 38923 } } peers { permanent_uuid: "3a856b77389a4578b24d490231334fbe" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 38419 } }
I20260810 23:05:31.162987 14316 heartbeater.cc:503] Master 127.15.56.62:40673 was elected leader, sending a full tablet report...
I20260810 23:05:31.163242 14559 ts_tablet_manager.cc:1428] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:05:31.163244 14558 ts_tablet_manager.cc:1428] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 23:05:31.163249 14562 ts_tablet_manager.cc:1428] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260810 23:05:31.163589 14106 heartbeater.cc:503] Master 127.15.56.62:40673 was elected leader, sending a full tablet report...
I20260810 23:05:31.163618 14531 heartbeater.cc:503] Master 127.15.56.62:40673 was elected leader, sending a full tablet report...
I20260810 23:05:31.289091 14571 raft_consensus.cc:482] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:31.289350 14571 raft_consensus.cc:504] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe487574e25b48c29a09551b86946296" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35441 } } peers { permanent_uuid: "6173bac0950347b69ebd912e51b7783d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 38923 } } peers { permanent_uuid: "3a856b77389a4578b24d490231334fbe" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 38419 } }
I20260810 23:05:31.290608 14571 leader_election.cc:288] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6173bac0950347b69ebd912e51b7783d (127.15.56.1:38923), 3a856b77389a4578b24d490231334fbe (127.15.56.3:38419)
I20260810 23:05:31.294719 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.291160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:48589 (local address 127.15.56.1:38923)
0810 23:05:31.291302 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:31.291307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:31.291509 (+ 202us) server_negotiation.cc:407] Connection header received
0810 23:05:31.291642 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:31.291645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:31.291748 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:31.291824 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:31.292147 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.292664 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.293769 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.294178 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.294283 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.294348 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:31.294499 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 23:05:31.294576 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 23:05:31.294721 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.290895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:38419 (local address 127.15.56.2:35669)
0810 23:05:31.291059 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:31.291076 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:31.291187 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:31.291473 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:31.291477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:31.291534 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:31.291694 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.291702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.292647 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.292650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.293521 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.293532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.294248 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.294252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.294370 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.294396 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:05:31.294461 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:05:31.294577 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":44,"spinlock_wait_cycles":8960}
I20260810 23:05:31.294716 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.291078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:35669 (local address 127.15.56.3:38419)
0810 23:05:31.291233 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:31.291237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:31.291253 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:05:31.291312 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:31.291315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:31.291363 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:31.291443 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:31.291861 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.292526 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.293660 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.294096 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.294192 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.294253 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:31.294499 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 23:05:31.294577 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 23:05:31.294750 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.290895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:38923 (local address 127.15.56.2:48589)
0810 23:05:31.291411 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:31.291424 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:31.291543 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:31.291875 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:31.291878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:31.291899 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:31.292032 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.292038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.292792 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.292795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.293631 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.293643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.294305 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.294308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.294384 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.294402 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 23:05:31.294470 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:05:31.294578 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":189,"spinlock_wait_cycles":36480,"thread_start_us":135,"threads_started":1}
I20260810 23:05:31.295564 14461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e5cb3d81a46db8171c74b82ba615e" candidate_uuid: "fe487574e25b48c29a09551b86946296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a856b77389a4578b24d490231334fbe" is_pre_election: true
I20260810 23:05:31.295588 14038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e5cb3d81a46db8171c74b82ba615e" candidate_uuid: "fe487574e25b48c29a09551b86946296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6173bac0950347b69ebd912e51b7783d" is_pre_election: true
I20260810 23:05:31.295918 14038 raft_consensus.cc:2454] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe487574e25b48c29a09551b86946296 in term 0.
I20260810 23:05:31.295918 14461 raft_consensus.cc:2454] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe487574e25b48c29a09551b86946296 in term 0.
I20260810 23:05:31.296430 14141 leader_election.cc:302] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173bac0950347b69ebd912e51b7783d, fe487574e25b48c29a09551b86946296; no voters:
I20260810 23:05:31.296726 14571 raft_consensus.cc:2781] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:05:31.296841 14571 raft_consensus.cc:482] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:05:31.296887 14571 raft_consensus.cc:3037] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:31.298036 14571 raft_consensus.cc:504] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe487574e25b48c29a09551b86946296" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35441 } } peers { permanent_uuid: "6173bac0950347b69ebd912e51b7783d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 38923 } } peers { permanent_uuid: "3a856b77389a4578b24d490231334fbe" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 38419 } }
I20260810 23:05:31.298460 14571 leader_election.cc:288] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [CANDIDATE]: Term 1 election: Requested vote from peers 6173bac0950347b69ebd912e51b7783d (127.15.56.1:38923), 3a856b77389a4578b24d490231334fbe (127.15.56.3:38419)
I20260810 23:05:31.298938 14038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e5cb3d81a46db8171c74b82ba615e" candidate_uuid: "fe487574e25b48c29a09551b86946296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6173bac0950347b69ebd912e51b7783d"
I20260810 23:05:31.299091 14038 raft_consensus.cc:3037] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:31.300321 14038 raft_consensus.cc:2454] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe487574e25b48c29a09551b86946296 in term 1.
I20260810 23:05:31.300632 14461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998e5cb3d81a46db8171c74b82ba615e" candidate_uuid: "fe487574e25b48c29a09551b86946296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a856b77389a4578b24d490231334fbe"
I20260810 23:05:31.300674 14141 leader_election.cc:302] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6173bac0950347b69ebd912e51b7783d, fe487574e25b48c29a09551b86946296; no voters:
I20260810 23:05:31.300770 14461 raft_consensus.cc:3037] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:31.300858 14571 raft_consensus.cc:2781] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:31.301122 14571 raft_consensus.cc:686] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 1 LEADER]: Becoming Leader. State: Replica: fe487574e25b48c29a09551b86946296, State: Running, Role: LEADER
I20260810 23:05:31.301534 14571 consensus_queue.cc:237] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe487574e25b48c29a09551b86946296" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35441 } } peers { permanent_uuid: "6173bac0950347b69ebd912e51b7783d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 38923 } } peers { permanent_uuid: "3a856b77389a4578b24d490231334fbe" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 38419 } }
I20260810 23:05:31.302227 14461 raft_consensus.cc:2454] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe487574e25b48c29a09551b86946296 in term 1.
I20260810 23:05:31.305408 13803 catalog_manager.cc:5495] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe487574e25b48c29a09551b86946296 (127.15.56.2). New cstate: current_term: 1 leader_uuid: "fe487574e25b48c29a09551b86946296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe487574e25b48c29a09551b86946296" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6173bac0950347b69ebd912e51b7783d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 38923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a856b77389a4578b24d490231334fbe" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 38419 } health_report { overall_health: UNKNOWN } } }
I20260810 23:05:31.334812 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.331045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.15.56.2:35441)
0810 23:05:31.331179 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:31.331184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:31.331199 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:05:31.331344 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:31.331346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:31.331412 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:31.331512 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:05:31.331842 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.332387 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.333258 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.333474 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.333566 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.333913 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:05:31.334390 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:05:31.334511 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:31.334589 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 23:05:31.334656 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 23:05:31.336804 14532 tablet.cc:2369] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:05:31.341533 14461 raft_consensus.cc:1261] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe [term 1 FOLLOWER]: Refusing update from remote peer fe487574e25b48c29a09551b86946296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:31.341533 14038 raft_consensus.cc:1261] T 998e5cb3d81a46db8171c74b82ba615e P 6173bac0950347b69ebd912e51b7783d [term 1 FOLLOWER]: Refusing update from remote peer fe487574e25b48c29a09551b86946296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:31.342237 14571 consensus_queue.cc:1035] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6173bac0950347b69ebd912e51b7783d" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:31.342465 14576 consensus_queue.cc:1035] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a856b77389a4578b24d490231334fbe" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 38419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:31.350353 14581 mvcc.cc:205] Tried to move back new op lower bound from 7317107225966981120 to 7317107225814839296. Current Snapshot: MvccSnapshot[applied={T|T < 7317107225966981120}]
I20260810 23:05:31.351689 14582 mvcc.cc:205] Tried to move back new op lower bound from 7317107225966981120 to 7317107225814839296. Current Snapshot: MvccSnapshot[applied={T|T < 7317107225966981120}]
I20260810 23:05:31.363883 13808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:31.364074 13808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:31.366776 13808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65fbac9f524945baad1ac94fcae5f7c7: Sending DeleteTablet for 3 replicas of tablet 998e5cb3d81a46db8171c74b82ba615e
I20260810 23:05:31.367717 14005 tablet_service.cc:1514] Processing DeleteTablet for tablet 998e5cb3d81a46db8171c74b82ba615e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:31 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:53182
I20260810 23:05:31.367836 14210 tablet_service.cc:1514] Processing DeleteTablet for tablet 998e5cb3d81a46db8171c74b82ba615e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:31 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:35536
I20260810 23:05:31.368459 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 13902
I20260810 23:05:31.368403 14426 tablet_service.cc:1514] Processing DeleteTablet for tablet 998e5cb3d81a46db8171c74b82ba615e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:31 UTC) from {username='kudu', principal='kudu/127.15.56.62@KRBTEST.COM'} at 127.0.0.1:35992
I20260810 23:05:31.369215 14600 tablet_replica.cc:330] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296: stopping tablet replica
I20260810 23:05:31.369553 14600 raft_consensus.cc:2229] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:05:31.369956 14600 raft_consensus.cc:2258] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:31.386402 14600 ts_tablet_manager.cc:1905] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:31.390048 14600 ts_tablet_manager.cc:1918] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:31.390218 14600 log.cc:1196] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/998e5cb3d81a46db8171c74b82ba615e
I20260810 23:05:31.390563 14600 ts_tablet_manager.cc:1939] T 998e5cb3d81a46db8171c74b82ba615e P fe487574e25b48c29a09551b86946296: Deleting consensus metadata
I20260810 23:05:31.392261 13784 catalog_manager.cc:4841] TS fe487574e25b48c29a09551b86946296 (127.15.56.2:35441): tablet 998e5cb3d81a46db8171c74b82ba615e (table test-table [id=a994e1fb050d4cae9685c2cb236700eb]) successfully deleted
I20260810 23:05:31.393234 14603 tablet_replica.cc:330] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: stopping tablet replica
I20260810 23:05:31.393640 14603 raft_consensus.cc:2229] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:31.394037 14603 raft_consensus.cc:2258] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:31.395783 14603 ts_tablet_manager.cc:1905] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:05:31.395799 13781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:38923 (error 108)
I20260810 23:05:31.395973 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 14112
I20260810 23:05:31.398407 14603 ts_tablet_manager.cc:1918] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:31.398555 14603 log.cc:1196] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/998e5cb3d81a46db8171c74b82ba615e
I20260810 23:05:31.398917 14603 ts_tablet_manager.cc:1939] T 998e5cb3d81a46db8171c74b82ba615e P 3a856b77389a4578b24d490231334fbe: Deleting consensus metadata
I20260810 23:05:31.400722 13784 catalog_manager.cc:4841] TS 3a856b77389a4578b24d490231334fbe (127.15.56.3:38419): tablet 998e5cb3d81a46db8171c74b82ba615e (table test-table [id=a994e1fb050d4cae9685c2cb236700eb]) successfully deleted
I20260810 23:05:31.401233 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.400921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:38923 (local address 127.0.0.1:53194)
0810 23:05:31.401052 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:31.401131 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:05:31.401425 13781 catalog_manager.cc:4584] TS 6173bac0950347b69ebd912e51b7783d (127.15.56.1:38923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 998e5cb3d81a46db8171c74b82ba615e: Network error: Client connection negotiation failed: client connection to 127.15.56.1:38923: connect: Connection refused (error 111)
I20260810 23:05:31.407886 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 14322
I20260810 23:05:31.422698 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 13758
2026-08-10T23:05:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2591 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T23:05:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:31Z Disabled control of system clock
I20260810 23:05:31.465832 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:34925
--webserver_interface=127.15.56.62
--webserver_port=0
--builtin_ntp_servers=127.15.56.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:34925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:05:31.583455 14610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:31.583818 14610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:31.583863 14610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:31.588037 14610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:31.588110 14610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:31.588124 14610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:31.588140 14610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:31.588155 14610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:31.593225 14610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:34925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:34925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:31.594465 14610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:31.595989 14610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:31.604876 14622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:31.604858 14619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:31.604854 14618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:31.605196 14610 server_base.cc:1034] running on GCE node
I20260810 23:05:31.605803 14610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:31.606947 14610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:31.608104 14610 hybrid_clock.cc:648] HybridClock initialized: now 1786403131608080 us; error 39 us; skew 500 ppm
I20260810 23:05:31.610047 14610 webserver.cc:460] Webserver started at http://127.15.56.62:36295/ using document root <none> and password file <none>
I20260810 23:05:31.610635 14610 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:31.610708 14610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:31.610930 14610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:31.613090 14610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "23acf1e0dfa048b0810afbba5f18c27c"
format_stamp: "Formatted at 2026-08-10 23:05:31 on dist-test-slave-6bbx"
server_key: "abc5adfc2a96acbefc764067024a80f3"
server_key_iv: "7b7bdd47937ba99667328672d905578e"
server_key_version: "encryptionkey@0"
I20260810 23:05:31.613632 14610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "23acf1e0dfa048b0810afbba5f18c27c"
format_stamp: "Formatted at 2026-08-10 23:05:31 on dist-test-slave-6bbx"
server_key: "abc5adfc2a96acbefc764067024a80f3"
server_key_iv: "7b7bdd47937ba99667328672d905578e"
server_key_version: "encryptionkey@0"
I20260810 23:05:31.617358 14610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:05:31.619979 14631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:31.621266 14610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:31.621397 14610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "23acf1e0dfa048b0810afbba5f18c27c"
format_stamp: "Formatted at 2026-08-10 23:05:31 on dist-test-slave-6bbx"
server_key: "abc5adfc2a96acbefc764067024a80f3"
server_key_iv: "7b7bdd47937ba99667328672d905578e"
server_key_version: "encryptionkey@0"
I20260810 23:05:31.621506 14610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:31.635612 14610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:31.636467 14610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:31.636668 14610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:31.643989 14610 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:34925
I20260810 23:05:31.644083 14720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:34925 every 8 connection(s)
I20260810 23:05:31.645032 14610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:31.648542 14722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:31.652631 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 14610
I20260810 23:05:31.652788 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:31.653120 15584 external_mini_cluster.cc:1428] Setting key 81ef87d600bc8694d65c6a4d2860aad9
I20260810 23:05:31.655576 14722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c: Bootstrap starting.
I20260810 23:05:31.658166 14722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:31.659005 14722 log.cc:824] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:31.661170 14722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c: No bootstrap required, opened a new log
I20260810 23:05:31.663487 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.654903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.15.56.62:34925)
0810 23:05:31.655830 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:31.655852 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:31.655898 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:05:31.656669 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:31.656691 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:31.657157 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:31.657520 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:31.658038 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.659101 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.660095 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.660602 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.660759 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.661631 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:31.661637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:31.661652 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:31.662078 (+ 426us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:31.662160 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:31.662183 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:31.662274 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:31.662435 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 23:05:31.662717 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":156,"threads_started":1}
I20260810 23:05:31.664978 14722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23acf1e0dfa048b0810afbba5f18c27c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 34925 } }
I20260810 23:05:31.665217 14722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:31.665257 14722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23acf1e0dfa048b0810afbba5f18c27c, State: Initialized, Role: FOLLOWER
I20260810 23:05:31.665719 14722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23acf1e0dfa048b0810afbba5f18c27c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 34925 } }
I20260810 23:05:31.665891 14722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:31.665963 14722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:31.666061 14722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:31.667011 14722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23acf1e0dfa048b0810afbba5f18c27c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 34925 } }
I20260810 23:05:31.667358 14722 leader_election.cc:302] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23acf1e0dfa048b0810afbba5f18c27c; no voters:
I20260810 23:05:31.667683 14722 leader_election.cc:288] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:31.667795 14728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:31.668009 14728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [term 1 LEADER]: Becoming Leader. State: Replica: 23acf1e0dfa048b0810afbba5f18c27c, State: Running, Role: LEADER
I20260810 23:05:31.668367 14728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23acf1e0dfa048b0810afbba5f18c27c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 34925 } }
I20260810 23:05:31.668932 14722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:31.670039 14730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23acf1e0dfa048b0810afbba5f18c27c. Latest consensus state: current_term: 1 leader_uuid: "23acf1e0dfa048b0810afbba5f18c27c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23acf1e0dfa048b0810afbba5f18c27c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 34925 } } }
I20260810 23:05:31.670177 14730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:31.670475 14729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23acf1e0dfa048b0810afbba5f18c27c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23acf1e0dfa048b0810afbba5f18c27c" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 34925 } } }
I20260810 23:05:31.670580 14729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:31.670665 14742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:31.673666 14742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:31.679759 14742 catalog_manager.cc:1347] Generated new cluster ID: 903ad35f3dcb464b87d613a582ac7fb9
I20260810 23:05:31.679847 14742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:31.699779 14742 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:31.700799 14742 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:31.706684 14742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c: Generated new TSK 0
I20260810 23:05:31.707520 14742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:31.718312 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.1:0
--local_ip_for_outbound_sockets=127.15.56.1
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:34925
--builtin_ntp_servers=127.15.56.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:05:31.840297 14762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:31.840600 14762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:31.840639 14762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:31.844767 14762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:31.844846 14762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:31.844903 14762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.1
I20260810 23:05:31.849772 14762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.56.1
--webserver_port=0
--tserver_master_addrs=127.15.56.62:34925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.1
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:31.851315 14762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:31.852840 14762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:05:31.863320 14762 server_base.cc:1034] running on GCE node
W20260810 23:05:31.863273 14772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:31.863273 14773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:31.863270 14776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:31.864005 14762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:31.864801 14762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:31.866006 14762 hybrid_clock.cc:648] HybridClock initialized: now 1786403131865966 us; error 45 us; skew 500 ppm
I20260810 23:05:31.868345 14762 webserver.cc:460] Webserver started at http://127.15.56.1:35593/ using document root <none> and password file <none>
I20260810 23:05:31.868992 14762 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:31.869071 14762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:31.869338 14762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:31.871855 14762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/data/instance:
uuid: "7b42062caf8546c3baa45f0a494f689b"
format_stamp: "Formatted at 2026-08-10 23:05:31 on dist-test-slave-6bbx"
server_key: "91c1225f83a63f6d6dfa84140977f165"
server_key_iv: "027f4bc80f96aa631ecec74cb011db97"
server_key_version: "encryptionkey@0"
I20260810 23:05:31.872512 14762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance:
uuid: "7b42062caf8546c3baa45f0a494f689b"
format_stamp: "Formatted at 2026-08-10 23:05:31 on dist-test-slave-6bbx"
server_key: "91c1225f83a63f6d6dfa84140977f165"
server_key_iv: "027f4bc80f96aa631ecec74cb011db97"
server_key_version: "encryptionkey@0"
I20260810 23:05:31.876827 14762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:05:31.879829 14783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:31.881381 14762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:05:31.881531 14762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/wal
uuid: "7b42062caf8546c3baa45f0a494f689b"
format_stamp: "Formatted at 2026-08-10 23:05:31 on dist-test-slave-6bbx"
server_key: "91c1225f83a63f6d6dfa84140977f165"
server_key_iv: "027f4bc80f96aa631ecec74cb011db97"
server_key_version: "encryptionkey@0"
I20260810 23:05:31.881657 14762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:31.911366 14762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:31.912312 14762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:31.912541 14762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:31.913235 14762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:31.914327 14762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:31.914388 14762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:31.914448 14762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:31.914476 14762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:31.923976 14762 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.1:39117
I20260810 23:05:31.924021 14946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.1:39117 every 8 connection(s)
I20260810 23:05:31.925041 14762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:31.926893 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 14762
I20260810 23:05:31.927022 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-0/wal/instance
I20260810 23:05:31.927306 15584 external_mini_cluster.cc:1428] Setting key bbeb0875a98c154747d0ae3e235ddb4f
I20260810 23:05:31.929368 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.2:0
--local_ip_for_outbound_sockets=127.15.56.2
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:34925
--builtin_ntp_servers=127.15.56.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:05:31.935415 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.927802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.1:37657 (local address 127.15.56.62:34925)
0810 23:05:31.927938 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:31.927943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:31.928790 (+ 847us) server_negotiation.cc:407] Connection header received
0810 23:05:31.930340 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:31.930344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:31.930406 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:31.930484 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:31.931499 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.932093 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.933342 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.933578 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.933671 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.934894 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:31.934898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:31.934900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:31.934914 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:31.934936 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:31.934944 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:31.935005 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:31.935177 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 23:05:31.935251 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:05:31.936465 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:31.927702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:34925 (local address 127.15.56.1:37657)
0810 23:05:31.928627 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:31.928690 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:31.930098 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:31.930697 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:31.930705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:31.930806 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:31.931336 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.931360 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.932241 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.932244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.933203 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:31.933214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:31.933748 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:31.933753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:31.933935 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:31.934584 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:05:31.934585 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:31.934637 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:05:31.934717 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:31.935019 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:31.935030 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:31.935038 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:05:31.935159 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:05:31.935501 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":163,"threads_started":1}
I20260810 23:05:31.937803 14949 heartbeater.cc:347] Connected to a master server at 127.15.56.62:34925
I20260810 23:05:31.938082 14949 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:31.938651 14949 heartbeater.cc:511] Master 127.15.56.62:34925 requested a full tablet report, sending...
I20260810 23:05:31.940445 14667 ts_manager.cc:194] Registered new tserver with Master: 7b42062caf8546c3baa45f0a494f689b (127.15.56.1:39117)
I20260810 23:05:31.942046 14667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.1:37657
W20260810 23:05:32.050501 14952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:32.050812 14952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:32.050863 14952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:32.055025 14952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:32.055104 14952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:32.055161 14952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.2
I20260810 23:05:32.060403 14952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.56.2
--webserver_port=0
--tserver_master_addrs=127.15.56.62:34925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.2
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:32.061549 14952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:32.063033 14952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:32.074096 14963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:32.074112 14962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:32.074105 14965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:32.074440 14952 server_base.cc:1034] running on GCE node
I20260810 23:05:32.074899 14952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:32.075691 14952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:32.076894 14952 hybrid_clock.cc:648] HybridClock initialized: now 1786403132076858 us; error 46 us; skew 500 ppm
I20260810 23:05:32.079140 14952 webserver.cc:460] Webserver started at http://127.15.56.2:33787/ using document root <none> and password file <none>
I20260810 23:05:32.079927 14952 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:32.080008 14952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:32.080230 14952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:32.082618 14952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/data/instance:
uuid: "e165c0b0cbf843f59adebac37c1ca79f"
format_stamp: "Formatted at 2026-08-10 23:05:32 on dist-test-slave-6bbx"
server_key: "6735ffbf9c18cad5095e84365fa2b8d1"
server_key_iv: "a63377136a8a27148844005a5476f43a"
server_key_version: "encryptionkey@0"
I20260810 23:05:32.083240 14952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance:
uuid: "e165c0b0cbf843f59adebac37c1ca79f"
format_stamp: "Formatted at 2026-08-10 23:05:32 on dist-test-slave-6bbx"
server_key: "6735ffbf9c18cad5095e84365fa2b8d1"
server_key_iv: "a63377136a8a27148844005a5476f43a"
server_key_version: "encryptionkey@0"
I20260810 23:05:32.087559 14952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 23:05:32.090490 14971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:32.091984 14952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:32.092106 14952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/wal
uuid: "e165c0b0cbf843f59adebac37c1ca79f"
format_stamp: "Formatted at 2026-08-10 23:05:32 on dist-test-slave-6bbx"
server_key: "6735ffbf9c18cad5095e84365fa2b8d1"
server_key_iv: "a63377136a8a27148844005a5476f43a"
server_key_version: "encryptionkey@0"
I20260810 23:05:32.092242 14952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:32.112295 14952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:32.113075 14952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:32.113266 14952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:32.113957 14952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:32.115033 14952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:32.115087 14952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:32.115131 14952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:32.115160 14952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:32.124294 14952 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.2:46661
I20260810 23:05:32.124323 15147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.2:46661 every 8 connection(s)
I20260810 23:05:32.125298 14952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:32.127202 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 14952
I20260810 23:05:32.127343 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/wal/instance
I20260810 23:05:32.127707 15584 external_mini_cluster.cc:1428] Setting key 4d1fd595b632e0ff2374ae1c758892fb
I20260810 23:05:32.130059 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.56.3:0
--local_ip_for_outbound_sockets=127.15.56.3
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:34925
--builtin_ntp_servers=127.15.56.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:05:32.136225 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.127363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.2:51329 (local address 127.15.56.62:34925)
0810 23:05:32.127524 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:32.127529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:32.128898 (+ 1369us) server_negotiation.cc:407] Connection header received
0810 23:05:32.130593 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:32.130597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:32.130658 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:32.130754 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:32.131814 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.132374 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.133861 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.134099 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.134193 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.135624 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:32.135628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:32.135630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:32.135644 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:32.135706 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:32.135718 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:32.135784 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:32.136021 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 23:05:32.136093 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 23:05:32.137555 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.127854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:34925 (local address 127.15.56.2:51329)
0810 23:05:32.128682 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:32.128775 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:32.130327 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:32.130948 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:32.130957 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:32.131059 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:32.131618 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.131633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.132521 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.132525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.133707 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.133722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.134288 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.134292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.134503 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.135259 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:05:32.135262 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:32.135333 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:05:32.135439 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:32.135802 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:32.135814 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:32.135821 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:05:32.135982 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:05:32.136368 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":129,"threads_started":1}
I20260810 23:05:32.138818 15149 heartbeater.cc:347] Connected to a master server at 127.15.56.62:34925
I20260810 23:05:32.139115 15149 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:32.139916 15149 heartbeater.cc:511] Master 127.15.56.62:34925 requested a full tablet report, sending...
I20260810 23:05:32.141155 14667 ts_manager.cc:194] Registered new tserver with Master: e165c0b0cbf843f59adebac37c1ca79f (127.15.56.2:46661)
I20260810 23:05:32.141742 14667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.2:51329
W20260810 23:05:32.254137 15155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:32.254458 15155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:32.254515 15155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:32.258647 15155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:32.258719 15155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:32.258795 15155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.56.3
I20260810 23:05:32.263866 15155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.56.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.56.3
--webserver_port=0
--tserver_master_addrs=127.15.56.62:34925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.56.3
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:32.265085 15155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:32.266638 15155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:05:32.277092 15155 server_base.cc:1034] running on GCE node
W20260810 23:05:32.277048 15163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:32.277055 15166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:32.277048 15164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:32.277750 15155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:32.278425 15155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:32.279584 15155 hybrid_clock.cc:648] HybridClock initialized: now 1786403132279550 us; error 44 us; skew 500 ppm
I20260810 23:05:32.281543 15155 webserver.cc:460] Webserver started at http://127.15.56.3:43459/ using document root <none> and password file <none>
I20260810 23:05:32.282155 15155 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:32.282223 15155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:32.282438 15155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:32.284672 15155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/data/instance:
uuid: "79964b0f2c654509885a396c5463286b"
format_stamp: "Formatted at 2026-08-10 23:05:32 on dist-test-slave-6bbx"
server_key: "a88b488188d230a219d71e46c911b233"
server_key_iv: "c6efed15bc2d15c19450148a14bc6e61"
server_key_version: "encryptionkey@0"
I20260810 23:05:32.285244 15155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance:
uuid: "79964b0f2c654509885a396c5463286b"
format_stamp: "Formatted at 2026-08-10 23:05:32 on dist-test-slave-6bbx"
server_key: "a88b488188d230a219d71e46c911b233"
server_key_iv: "c6efed15bc2d15c19450148a14bc6e61"
server_key_version: "encryptionkey@0"
I20260810 23:05:32.289155 15155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:32.291785 15180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:32.293115 15155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:32.293226 15155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/wal
uuid: "79964b0f2c654509885a396c5463286b"
format_stamp: "Formatted at 2026-08-10 23:05:32 on dist-test-slave-6bbx"
server_key: "a88b488188d230a219d71e46c911b233"
server_key_iv: "c6efed15bc2d15c19450148a14bc6e61"
server_key_version: "encryptionkey@0"
I20260810 23:05:32.293344 15155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:32.306756 15155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:32.307549 15155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:32.307817 15155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:32.308499 15155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:32.309563 15155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:32.309614 15155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:32.309659 15155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:32.309691 15155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:32.319053 15155 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.3:42219
I20260810 23:05:32.319128 15354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.3:42219 every 8 connection(s)
I20260810 23:05:32.320118 15155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/data/info.pb
I20260810 23:05:32.328701 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 15155
I20260810 23:05:32.328855 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/wal/instance
I20260810 23:05:32.329171 15584 external_mini_cluster.cc:1428] Setting key 82a162aba2f81a8833fd346ce33b9819
I20260810 23:05:32.329977 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.322116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:37205 (local address 127.15.56.62:34925)
0810 23:05:32.322288 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:32.322292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:32.323469 (+ 1177us) server_negotiation.cc:407] Connection header received
0810 23:05:32.325043 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:32.325047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:32.325106 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:32.325189 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:32.326177 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.326724 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.327951 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.328184 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.328282 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.329480 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:32.329484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:32.329487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:32.329504 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:32.329527 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:32.329536 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:32.329596 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:32.329756 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 23:05:32.329828 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 23:05:32.330992 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.322555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:34925 (local address 127.15.56.3:37205)
0810 23:05:32.323299 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:32.323349 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:32.324796 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:32.325396 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:32.325404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:32.325497 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:32.326039 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.326059 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.326858 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.326861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.327833 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.327844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.328345 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.328349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.328532 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.329188 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:05:32.329190 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:32.329236 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:05:32.329325 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:32.329606 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:32.329616 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:32.329622 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:05:32.329749 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 23:05:32.330074 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":122,"threads_started":1}
I20260810 23:05:32.332041 15355 heartbeater.cc:347] Connected to a master server at 127.15.56.62:34925
I20260810 23:05:32.332347 15355 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:32.332907 15355 heartbeater.cc:511] Master 127.15.56.62:34925 requested a full tablet report, sending...
I20260810 23:05:32.333945 14667 ts_manager.cc:194] Registered new tserver with Master: 79964b0f2c654509885a396c5463286b (127.15.56.3:42219)
I20260810 23:05:32.334539 14667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.56.3:37205
I20260810 23:05:32.343642 15584 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:05:32.349933 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.345782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.15.56.62:34925)
0810 23:05:32.345928 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:32.345933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:32.346095 (+ 162us) server_negotiation.cc:407] Connection header received
0810 23:05:32.346274 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:32.346277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:32.346328 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:32.346405 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:32.346748 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.347316 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.348224 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.348446 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.348529 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.349377 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:32.349381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:32.349383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:32.349397 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:32.349421 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:32.349429 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:32.349503 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:32.349687 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 23:05:32.349759 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:05:32.353197 14667 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:05:32.356477 14667 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:05:32.372323 15380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.366460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:46661 (local address 127.0.0.1:36830)
0810 23:05:32.367502 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:32.367513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:32.367608 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:32.368096 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:32.368099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:32.368119 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:32.368213 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.368219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.369224 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.369228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.370792 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.370800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.371693 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.371697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.371818 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.371860 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 23:05:32.371996 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 23:05:32.372143 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":8,"thread_start_us":86,"threads_started":1}
I20260810 23:05:32.372327 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.366901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.15.56.2:46661)
0810 23:05:32.367365 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:32.367376 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:32.367607 (+ 231us) server_negotiation.cc:407] Connection header received
0810 23:05:32.367745 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:32.367751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:32.367959 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:32.368112 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:32.368318 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.369111 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.370913 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.371534 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.371639 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.371922 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:32.372031 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 23:05:32.372143 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":126,"threads_started":1}
I20260810 23:05:32.372588 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.365608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.3:42219 (local address 127.0.0.1:50714)
0810 23:05:32.366544 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:32.366608 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:32.366794 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:32.367540 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:32.367548 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:32.367587 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:32.367853 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.367859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.369448 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.369452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.370602 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.370614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.372034 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.372038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.372221 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.372270 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 23:05:32.372342 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 23:05:32.372425 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":14,"spinlock_wait_cycles":6272,"thread_start_us":111,"threads_started":1}
I20260810 23:05:32.373054 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.365778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.15.56.3:42219)
0810 23:05:32.366480 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:32.366492 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:32.366711 (+ 219us) server_negotiation.cc:407] Connection header received
0810 23:05:32.366989 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:32.366995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:32.367217 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:32.367357 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:32.367986 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.369258 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.370738 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.371622 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.371758 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.371966 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:32.372748 (+ 782us) server_negotiation.cc:299] Negotiation successful
0810 23:05:32.372880 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":101,"threads_started":1}
I20260810 23:05:32.373364 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.365771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:39117 (local address 127.0.0.1:52354)
0810 23:05:32.366626 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:32.366639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:32.366794 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:32.367540 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:32.367547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:32.367585 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:32.367797 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.367806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.369385 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.369388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.370721 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.370733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.372034 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.372038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.372243 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.372277 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:05:32.372342 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:05:32.373234 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":837,"spinlock_wait_cycles":8064,"thread_start_us":108,"threads_started":1}
I20260810 23:05:32.373874 15379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.366017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52354 (local address 127.15.56.1:39117)
0810 23:05:32.366645 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:32.366658 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:32.366938 (+ 280us) server_negotiation.cc:407] Connection header received
0810 23:05:32.367019 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:32.367023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:32.367217 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:32.367357 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:32.367939 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.369258 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.370885 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.371622 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.371758 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.371966 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:32.373574 (+ 1608us) server_negotiation.cc:299] Negotiation successful
0810 23:05:32.373711 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":152,"threads_started":1}
I20260810 23:05:32.375975 15049 tablet_service.cc:1467] Processing CreateTablet for tablet fc3a8f8c87fc457e8769fd6ee61634db (DEFAULT_TABLE table=test-table [id=a01cfd71fe8a4f15991829e686066885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:32.375975 15253 tablet_service.cc:1467] Processing CreateTablet for tablet fc3a8f8c87fc457e8769fd6ee61634db (DEFAULT_TABLE table=test-table [id=a01cfd71fe8a4f15991829e686066885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:32.377288 15253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3a8f8c87fc457e8769fd6ee61634db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:32.377288 15049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3a8f8c87fc457e8769fd6ee61634db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:32.379042 14857 tablet_service.cc:1467] Processing CreateTablet for tablet fc3a8f8c87fc457e8769fd6ee61634db (DEFAULT_TABLE table=test-table [id=a01cfd71fe8a4f15991829e686066885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:32.380172 14857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3a8f8c87fc457e8769fd6ee61634db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:32.386533 15383 tablet_bootstrap.cc:492] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b: Bootstrap starting.
I20260810 23:05:32.385947 15382 tablet_bootstrap.cc:492] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f: Bootstrap starting.
I20260810 23:05:32.388108 15386 tablet_bootstrap.cc:492] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b: Bootstrap starting.
I20260810 23:05:32.390501 15382 tablet_bootstrap.cc:654] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:32.390654 15383 tablet_bootstrap.cc:654] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:32.390653 15386 tablet_bootstrap.cc:654] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:32.391434 15382 log.cc:824] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:32.391618 15386 log.cc:824] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:32.391618 15383 log.cc:824] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:32.393945 15383 tablet_bootstrap.cc:492] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b: No bootstrap required, opened a new log
I20260810 23:05:32.394001 15382 tablet_bootstrap.cc:492] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f: No bootstrap required, opened a new log
I20260810 23:05:32.394001 15386 tablet_bootstrap.cc:492] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b: No bootstrap required, opened a new log
I20260810 23:05:32.394193 15383 ts_tablet_manager.cc:1397] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 23:05:32.394199 15382 ts_tablet_manager.cc:1397] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 23:05:32.394199 15386 ts_tablet_manager.cc:1397] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:05:32.398748 15383 raft_consensus.cc:348] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79964b0f2c654509885a396c5463286b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42219 } } peers { permanent_uuid: "7b42062caf8546c3baa45f0a494f689b" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 39117 } } peers { permanent_uuid: "e165c0b0cbf843f59adebac37c1ca79f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 46661 } }
I20260810 23:05:32.399039 15383 raft_consensus.cc:374] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:32.399096 15383 raft_consensus.cc:729] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79964b0f2c654509885a396c5463286b, State: Initialized, Role: FOLLOWER
I20260810 23:05:32.398937 15386 raft_consensus.cc:348] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79964b0f2c654509885a396c5463286b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42219 } } peers { permanent_uuid: "7b42062caf8546c3baa45f0a494f689b" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 39117 } } peers { permanent_uuid: "e165c0b0cbf843f59adebac37c1ca79f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 46661 } }
I20260810 23:05:32.398937 15382 raft_consensus.cc:348] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79964b0f2c654509885a396c5463286b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42219 } } peers { permanent_uuid: "7b42062caf8546c3baa45f0a494f689b" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 39117 } } peers { permanent_uuid: "e165c0b0cbf843f59adebac37c1ca79f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 46661 } }
I20260810 23:05:32.399215 15386 raft_consensus.cc:374] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:32.399215 15382 raft_consensus.cc:374] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:32.399288 15382 raft_consensus.cc:729] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e165c0b0cbf843f59adebac37c1ca79f, State: Initialized, Role: FOLLOWER
I20260810 23:05:32.399302 15386 raft_consensus.cc:729] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b42062caf8546c3baa45f0a494f689b, State: Initialized, Role: FOLLOWER
I20260810 23:05:32.399642 15383 consensus_queue.cc:260] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79964b0f2c654509885a396c5463286b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42219 } } peers { permanent_uuid: "7b42062caf8546c3baa45f0a494f689b" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 39117 } } peers { permanent_uuid: "e165c0b0cbf843f59adebac37c1ca79f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 46661 } }
I20260810 23:05:32.399791 15386 consensus_queue.cc:260] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79964b0f2c654509885a396c5463286b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42219 } } peers { permanent_uuid: "7b42062caf8546c3baa45f0a494f689b" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 39117 } } peers { permanent_uuid: "e165c0b0cbf843f59adebac37c1ca79f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 46661 } }
I20260810 23:05:32.399791 15382 consensus_queue.cc:260] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79964b0f2c654509885a396c5463286b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42219 } } peers { permanent_uuid: "7b42062caf8546c3baa45f0a494f689b" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 39117 } } peers { permanent_uuid: "e165c0b0cbf843f59adebac37c1ca79f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 46661 } }
I20260810 23:05:32.400494 15355 heartbeater.cc:503] Master 127.15.56.62:34925 was elected leader, sending a full tablet report...
I20260810 23:05:32.400935 15383 ts_tablet_manager.cc:1428] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260810 23:05:32.401021 15386 ts_tablet_manager.cc:1428] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260810 23:05:32.401410 14949 heartbeater.cc:503] Master 127.15.56.62:34925 was elected leader, sending a full tablet report...
I20260810 23:05:32.401558 15382 ts_tablet_manager.cc:1428] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 23:05:32.401887 15149 heartbeater.cc:503] Master 127.15.56.62:34925 was elected leader, sending a full tablet report...
I20260810 23:05:32.425027 15392 raft_consensus.cc:482] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:32.425328 15392 raft_consensus.cc:504] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79964b0f2c654509885a396c5463286b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42219 } } peers { permanent_uuid: "7b42062caf8546c3baa45f0a494f689b" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 39117 } } peers { permanent_uuid: "e165c0b0cbf843f59adebac37c1ca79f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 46661 } }
W20260810 23:05:32.426357 14950 tablet.cc:2369] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:05:32.426638 15392 leader_election.cc:288] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b42062caf8546c3baa45f0a494f689b (127.15.56.1:39117), e165c0b0cbf843f59adebac37c1ca79f (127.15.56.2:46661)
I20260810 23:05:32.430577 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.426916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:39117 (local address 127.15.56.3:57247)
0810 23:05:32.427057 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:32.427074 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:32.427178 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:32.427472 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:32.427476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:32.427499 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:32.427624 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.427632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.428586 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.428589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.429514 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.429525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.430161 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.430165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.430274 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.430294 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 23:05:32.430360 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 23:05:32.430433 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 23:05:32.430577 15379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.427017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:57247 (local address 127.15.56.1:39117)
0810 23:05:32.427163 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:32.427167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:32.427187 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:05:32.427283 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:32.427286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:32.427343 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:32.427428 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:32.427767 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.428489 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.429638 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.430045 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.430138 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.430204 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:32.430386 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 23:05:32.430460 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:05:32.430979 15381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.427357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.56.3:49877 (local address 127.15.56.2:46661)
0810 23:05:32.427477 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:32.427481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:32.427810 (+ 329us) server_negotiation.cc:407] Connection header received
0810 23:05:32.427937 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:32.427940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:32.427989 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:32.428060 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:32.428378 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.429008 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.430018 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.430406 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.430497 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.430559 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:32.430778 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 23:05:32.430859 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:05:32.431330 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.427235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.2:46661 (local address 127.15.56.3:49877)
0810 23:05:32.427722 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:32.427736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:32.427832 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:32.428082 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:32.428086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:32.428105 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:32.428249 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.428256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.429118 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.429121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.429891 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:32.429901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.430528 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.430532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:32.430628 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.430649 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:05:32.431134 (+ 485us) client_negotiation.cc:240] Negotiation successful
0810 23:05:32.431221 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":150,"threads_started":1}
I20260810 23:05:32.431293 14883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3a8f8c87fc457e8769fd6ee61634db" candidate_uuid: "79964b0f2c654509885a396c5463286b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b42062caf8546c3baa45f0a494f689b" is_pre_election: true
I20260810 23:05:32.431627 14883 raft_consensus.cc:2454] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79964b0f2c654509885a396c5463286b in term 0.
I20260810 23:05:32.432019 15080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3a8f8c87fc457e8769fd6ee61634db" candidate_uuid: "79964b0f2c654509885a396c5463286b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e165c0b0cbf843f59adebac37c1ca79f" is_pre_election: true
I20260810 23:05:32.432263 15080 raft_consensus.cc:2454] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79964b0f2c654509885a396c5463286b in term 0.
I20260810 23:05:32.432235 15185 leader_election.cc:302] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79964b0f2c654509885a396c5463286b, 7b42062caf8546c3baa45f0a494f689b; no voters:
I20260810 23:05:32.432538 15392 raft_consensus.cc:2781] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:05:32.432622 15392 raft_consensus.cc:482] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:05:32.432652 15392 raft_consensus.cc:3037] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:32.433717 15392 raft_consensus.cc:504] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79964b0f2c654509885a396c5463286b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42219 } } peers { permanent_uuid: "7b42062caf8546c3baa45f0a494f689b" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 39117 } } peers { permanent_uuid: "e165c0b0cbf843f59adebac37c1ca79f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 46661 } }
I20260810 23:05:32.434060 15392 leader_election.cc:288] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [CANDIDATE]: Term 1 election: Requested vote from peers 7b42062caf8546c3baa45f0a494f689b (127.15.56.1:39117), e165c0b0cbf843f59adebac37c1ca79f (127.15.56.2:46661)
I20260810 23:05:32.434476 14883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3a8f8c87fc457e8769fd6ee61634db" candidate_uuid: "79964b0f2c654509885a396c5463286b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b42062caf8546c3baa45f0a494f689b"
I20260810 23:05:32.434562 15080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3a8f8c87fc457e8769fd6ee61634db" candidate_uuid: "79964b0f2c654509885a396c5463286b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e165c0b0cbf843f59adebac37c1ca79f"
I20260810 23:05:32.434612 14883 raft_consensus.cc:3037] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:32.434713 15080 raft_consensus.cc:3037] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:32.435827 14883 raft_consensus.cc:2454] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79964b0f2c654509885a396c5463286b in term 1.
I20260810 23:05:32.435832 15080 raft_consensus.cc:2454] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79964b0f2c654509885a396c5463286b in term 1.
I20260810 23:05:32.436141 15185 leader_election.cc:302] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79964b0f2c654509885a396c5463286b, 7b42062caf8546c3baa45f0a494f689b; no voters:
I20260810 23:05:32.436324 15392 raft_consensus.cc:2781] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:32.436566 15392 raft_consensus.cc:686] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 1 LEADER]: Becoming Leader. State: Replica: 79964b0f2c654509885a396c5463286b, State: Running, Role: LEADER
I20260810 23:05:32.436942 15392 consensus_queue.cc:237] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79964b0f2c654509885a396c5463286b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42219 } } peers { permanent_uuid: "7b42062caf8546c3baa45f0a494f689b" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 39117 } } peers { permanent_uuid: "e165c0b0cbf843f59adebac37c1ca79f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 46661 } }
I20260810 23:05:32.440284 14663 catalog_manager.cc:5495] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b reported cstate change: term changed from 0 to 1, leader changed from <none> to 79964b0f2c654509885a396c5463286b (127.15.56.3). New cstate: current_term: 1 leader_uuid: "79964b0f2c654509885a396c5463286b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79964b0f2c654509885a396c5463286b" member_type: VOTER last_known_addr { host: "127.15.56.3" port: 42219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b42062caf8546c3baa45f0a494f689b" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 39117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e165c0b0cbf843f59adebac37c1ca79f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 46661 } health_report { overall_health: UNKNOWN } } }
I20260810 23:05:32.453295 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.449699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.15.56.3:42219)
0810 23:05:32.449827 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:32.449831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:32.449844 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:05:32.449936 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:32.449938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:32.449993 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:32.450095 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:05:32.450430 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.450938 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.451832 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:32.452041 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:32.452115 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:32.452437 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:05:32.452880 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:05:32.452970 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:32.453044 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 23:05:32.453092 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:05:32.460173 15080 raft_consensus.cc:1261] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f [term 1 FOLLOWER]: Refusing update from remote peer 79964b0f2c654509885a396c5463286b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:32.460170 14883 raft_consensus.cc:1261] T fc3a8f8c87fc457e8769fd6ee61634db P 7b42062caf8546c3baa45f0a494f689b [term 1 FOLLOWER]: Refusing update from remote peer 79964b0f2c654509885a396c5463286b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:32.460995 15396 consensus_queue.cc:1035] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e165c0b0cbf843f59adebac37c1ca79f" member_type: VOTER last_known_addr { host: "127.15.56.2" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:32.461232 15392 consensus_queue.cc:1035] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b42062caf8546c3baa45f0a494f689b" member_type: VOTER last_known_addr { host: "127.15.56.1" port: 39117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:32.470665 15402 mvcc.cc:205] Tried to move back new op lower bound from 7317107230547636224 to 7317107230464589824. Current Snapshot: MvccSnapshot[applied={T|T < 7317107230547636224}]
I20260810 23:05:32.471606 15406 mvcc.cc:205] Tried to move back new op lower bound from 7317107230547636224 to 7317107230464589824. Current Snapshot: MvccSnapshot[applied={T|T < 7317107230547636224}]
I20260810 23:05:32.477876 14667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:32.478094 14667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:32.481088 14667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23acf1e0dfa048b0810afbba5f18c27c: Sending DeleteTablet for 3 replicas of tablet fc3a8f8c87fc457e8769fd6ee61634db
I20260810 23:05:32.482028 15253 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3a8f8c87fc457e8769fd6ee61634db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:32 UTC) from {username='slave'} at 127.0.0.1:50714
I20260810 23:05:32.482148 15049 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3a8f8c87fc457e8769fd6ee61634db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:32 UTC) from {username='slave'} at 127.0.0.1:36830
I20260810 23:05:32.482674 14857 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3a8f8c87fc457e8769fd6ee61634db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:32 UTC) from {username='slave'} at 127.0.0.1:52354
I20260810 23:05:32.482791 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 14762
I20260810 23:05:32.482934 15417 tablet_replica.cc:330] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f: stopping tablet replica
I20260810 23:05:32.483372 15417 raft_consensus.cc:2229] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:32.483845 15417 raft_consensus.cc:2258] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:32.484262 15416 tablet_replica.cc:330] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b: stopping tablet replica
I20260810 23:05:32.484596 15416 raft_consensus.cc:2229] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:05:32.485024 15416 raft_consensus.cc:2258] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:32.487454 15416 ts_tablet_manager.cc:1905] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:32.490679 15416 ts_tablet_manager.cc:1918] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:32.490896 15416 log.cc:1196] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-2/wal/wals/fc3a8f8c87fc457e8769fd6ee61634db
I20260810 23:05:32.491381 15416 ts_tablet_manager.cc:1939] T fc3a8f8c87fc457e8769fd6ee61634db P 79964b0f2c654509885a396c5463286b: Deleting consensus metadata
I20260810 23:05:32.493124 14634 catalog_manager.cc:4841] TS 79964b0f2c654509885a396c5463286b (127.15.56.3:42219): tablet fc3a8f8c87fc457e8769fd6ee61634db (table test-table [id=a01cfd71fe8a4f15991829e686066885]) successfully deleted
I20260810 23:05:32.496220 15417 ts_tablet_manager.cc:1905] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:32.499245 15417 ts_tablet_manager.cc:1918] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:32.499370 15417 log.cc:1196] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f: Deleting WAL directory at /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402867637343-15584-0/minicluster-data/ts-1/wal/wals/fc3a8f8c87fc457e8769fd6ee61634db
I20260810 23:05:32.499656 15417 ts_tablet_manager.cc:1939] T fc3a8f8c87fc457e8769fd6ee61634db P e165c0b0cbf843f59adebac37c1ca79f: Deleting consensus metadata
I20260810 23:05:32.501655 14637 catalog_manager.cc:4841] TS e165c0b0cbf843f59adebac37c1ca79f (127.15.56.2:46661): tablet fc3a8f8c87fc457e8769fd6ee61634db (table test-table [id=a01cfd71fe8a4f15991829e686066885]) successfully deleted
W20260810 23:05:32.505482 14637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.56.1:39117 (error 108)
I20260810 23:05:32.505532 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 14952
I20260810 23:05:32.514169 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:32.513605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.1:39117 (local address 127.0.0.1:52366)
0810 23:05:32.513913 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:32.514034 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.1:39117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
W20260810 23:05:32.514434 14637 catalog_manager.cc:4584] TS 7b42062caf8546c3baa45f0a494f689b (127.15.56.1:39117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc3a8f8c87fc457e8769fd6ee61634db: Network error: Client connection negotiation failed: client connection to 127.15.56.1:39117: connect: Connection refused (error 111)
I20260810 23:05:32.538328 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 15155
I20260810 23:05:32.558276 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 14610
2026-08-10T23:05:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1136 ms)
[----------] 6 tests from EncryptionPolicyTest (11720 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:32 dist-test-slave-6bbx krb5kdc[15421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:32 dist-test-slave-6bbx krb5kdc[15421](info): set up 2 sockets
Aug 10 23:05:32 dist-test-slave-6bbx krb5kdc[15421](info): commencing operation
krb5kdc: starting...
W20260810 23:05:33.731566 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:33 dist-test-slave-6bbx krb5kdc[15421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:33.865185 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:33.984303 15439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:33.984611 15439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:33.984671 15439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:33.988818 15439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:33.988891 15439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:33.988917 15439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:33.988938 15439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:33.988960 15439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:33.994480 15439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39865
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:33.995599 15439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:33.996688 15439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:34.005447 15453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:34.005465 15451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:34.005465 15449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:34.005491 15439 server_base.cc:1034] running on GCE node
I20260810 23:05:34.006340 15439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:34.007467 15439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:34.008646 15439 hybrid_clock.cc:648] HybridClock initialized: now 1786403134008615 us; error 41 us; skew 500 ppm
Aug 10 23:05:34 dist-test-slave-6bbx krb5kdc[15421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:34.011437 15439 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:34.012383 15439 webserver.cc:460] Webserver started at http://127.0.0.1:40003/ using document root <none> and password file <none>
I20260810 23:05:34.012933 15439 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:34.013008 15439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:34.013213 15439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:34.015290 15439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "731da7905ab44c699bc82c476c649e40"
format_stamp: "Formatted at 2026-08-10 23:05:34 on dist-test-slave-6bbx"
server_key: "7e7bb401b177afea66a75dcf77b614dd"
server_key_iv: "b39be771122a010c5c48c3cb92ed956d"
server_key_version: "encryptionkey@0"
I20260810 23:05:34.015946 15439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "731da7905ab44c699bc82c476c649e40"
format_stamp: "Formatted at 2026-08-10 23:05:34 on dist-test-slave-6bbx"
server_key: "7e7bb401b177afea66a75dcf77b614dd"
server_key_iv: "b39be771122a010c5c48c3cb92ed956d"
server_key_version: "encryptionkey@0"
I20260810 23:05:34.019729 15439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260810 23:05:34.022218 15468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:34.023572 15439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:05:34.023731 15439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "731da7905ab44c699bc82c476c649e40"
format_stamp: "Formatted at 2026-08-10 23:05:34 on dist-test-slave-6bbx"
server_key: "7e7bb401b177afea66a75dcf77b614dd"
server_key_iv: "b39be771122a010c5c48c3cb92ed956d"
server_key_version: "encryptionkey@0"
I20260810 23:05:34.023849 15439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:34.048708 15439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:34.051898 15439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:34.052114 15439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:34.059571 15439 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39865
I20260810 23:05:34.059649 15554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39865 every 8 connection(s)
I20260810 23:05:34.060631 15439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:34.063299 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 15439
I20260810 23:05:34.063449 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:34.063772 15584 external_mini_cluster.cc:1428] Setting key 54519e2b9b5d85c04c8d77e55d9c3ef7
I20260810 23:05:34.063803 15555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:34.071652 15555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40: Bootstrap starting.
Aug 10 23:05:34 dist-test-slave-6bbx krb5kdc[15421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:05:34.074322 15555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:34.075332 15555 log.cc:824] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:34.077927 15555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40: No bootstrap required, opened a new log
I20260810 23:05:34.080300 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:34.065376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37229 (local address 127.0.0.1:39865)
0810 23:05:34.065849 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:34.065869 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:34.065916 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:05:34.066748 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:34.066775 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:34.067252 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:34.067691 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:34.068444 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:34.069954 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:34.070933 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:34.071743 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:34.071927 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:34.074695 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:34.074702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:34.074722 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:34.074762 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:34.076683 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:34.077296 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:34.077303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:34.077309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:34.077459 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:34.077862 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:34.077865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:34.077867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:34.078431 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:34.078676 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:34.078973 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 23:05:34.079228 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":127,"threads_started":1}
I20260810 23:05:34.082403 15555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731da7905ab44c699bc82c476c649e40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39865 } }
I20260810 23:05:34.082680 15555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:34.082743 15555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731da7905ab44c699bc82c476c649e40, State: Initialized, Role: FOLLOWER
I20260810 23:05:34.083184 15555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731da7905ab44c699bc82c476c649e40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39865 } }
I20260810 23:05:34.083321 15555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:34.083384 15555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:34.083488 15555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:34.083616 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:34.084769 15555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731da7905ab44c699bc82c476c649e40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39865 } }
I20260810 23:05:34.085199 15555 leader_election.cc:302] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 731da7905ab44c699bc82c476c649e40; no voters:
I20260810 23:05:34.085592 15555 leader_election.cc:288] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:34.085675 15574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:34.085919 15574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [term 1 LEADER]: Becoming Leader. State: Replica: 731da7905ab44c699bc82c476c649e40, State: Running, Role: LEADER
I20260810 23:05:34.086263 15574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731da7905ab44c699bc82c476c649e40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39865 } }
I20260810 23:05:34.087064 15555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:34.087927 15575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "731da7905ab44c699bc82c476c649e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731da7905ab44c699bc82c476c649e40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39865 } } }
I20260810 23:05:34.088059 15575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:34.088312 15575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 731da7905ab44c699bc82c476c649e40. Latest consensus state: current_term: 1 leader_uuid: "731da7905ab44c699bc82c476c649e40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731da7905ab44c699bc82c476c649e40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39865 } } }
I20260810 23:05:34.088402 15575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:34.088868 15596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:34.092790 15596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:34.095220 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:34.085985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45499 (local address 127.0.0.1:39865)
0810 23:05:34.086094 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:34.086098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:34.086332 (+ 234us) server_negotiation.cc:407] Connection header received
0810 23:05:34.086499 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:34.086503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:34.086556 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:34.086660 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:34.087458 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:34.088260 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:34.089401 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:34.089640 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:34.089726 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:34.091409 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:34.091413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:34.091416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:34.091442 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:34.093205 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:34.093767 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:34.093771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:34.093772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:34.093856 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:34.094224 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:34.094227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:34.094229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:34.094445 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:34.094585 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:34.094957 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 23:05:34.095059 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:05:34.099134 15596 catalog_manager.cc:1347] Generated new cluster ID: 0c4198db262d4f7cb3b5c5c5dc1f6856
I20260810 23:05:34.099207 15596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:34.114478 15596 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:34.115507 15596 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:34.119590 15596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 731da7905ab44c699bc82c476c649e40: Generated new TSK 0
I20260810 23:05:34.120419 15596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:34.125000 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 15439
W20260810 23:05:34.144641 15419 connection.cc:535] client connection to 127.0.0.1:39865 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39865): Connection reset by peer (error 104)
2026-08-10T23:05:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1567 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:34 dist-test-slave-6bbx krb5kdc[15616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:34 dist-test-slave-6bbx krb5kdc[15616](info): set up 2 sockets
Aug 10 23:05:34 dist-test-slave-6bbx krb5kdc[15616](info): commencing operation
krb5kdc: starting...
W20260810 23:05:35.311470 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:35 dist-test-slave-6bbx krb5kdc[15616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:35.447010 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:35.565245 15634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:35.565544 15634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:35.565605 15634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:35.569734 15634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:35.569811 15634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:35.569839 15634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:35.569859 15634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:35.569880 15634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:35.575325 15634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:35.576488 15634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:35.577564 15634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:35.586472 15643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:35.586472 15642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:35.586465 15647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:35.586593 15634 server_base.cc:1034] running on GCE node
I20260810 23:05:35.587337 15634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:35.588546 15634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:35.589721 15634 hybrid_clock.cc:648] HybridClock initialized: now 1786403135589702 us; error 28 us; skew 500 ppm
Aug 10 23:05:35 dist-test-slave-6bbx krb5kdc[15616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:35.596841 15634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:35.597858 15634 webserver.cc:460] Webserver started at http://127.0.0.1:42831/ using document root <none> and password file <none>
I20260810 23:05:35.598438 15634 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:35.598517 15634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:35.598749 15634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:35.600951 15634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "cbbd69ceedb84a5188e6e98b52f2885b"
format_stamp: "Formatted at 2026-08-10 23:05:35 on dist-test-slave-6bbx"
server_key: "86db2fc706cd8a38b47c8cacb9335891"
server_key_iv: "59fcf00ac94ebe71fec36589fff09e10"
server_key_version: "encryptionkey@0"
I20260810 23:05:35.601500 15634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "cbbd69ceedb84a5188e6e98b52f2885b"
format_stamp: "Formatted at 2026-08-10 23:05:35 on dist-test-slave-6bbx"
server_key: "86db2fc706cd8a38b47c8cacb9335891"
server_key_iv: "59fcf00ac94ebe71fec36589fff09e10"
server_key_version: "encryptionkey@0"
I20260810 23:05:35.605327 15634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:05:35.607885 15660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:35.609299 15634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:35.609442 15634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "cbbd69ceedb84a5188e6e98b52f2885b"
format_stamp: "Formatted at 2026-08-10 23:05:35 on dist-test-slave-6bbx"
server_key: "86db2fc706cd8a38b47c8cacb9335891"
server_key_iv: "59fcf00ac94ebe71fec36589fff09e10"
server_key_version: "encryptionkey@0"
I20260810 23:05:35.609557 15634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:35.628278 15634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:35.631733 15634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:35.631934 15634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:35.639274 15634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35113
I20260810 23:05:35.639253 15746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35113 every 8 connection(s)
I20260810 23:05:35.640394 15634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:35.643453 15748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:35.645123 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 15634
I20260810 23:05:35.645283 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:35.645612 15584 external_mini_cluster.cc:1428] Setting key acf105ed2ce7a0129e56a686931972bb
I20260810 23:05:35.650092 15748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b: Bootstrap starting.
I20260810 23:05:35.652479 15748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:35.653206 15748 log.cc:824] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:35.654772 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:35.647081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34125 (local address 127.0.0.1:35113)
0810 23:05:35.647587 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:35.647604 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:35.647644 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:05:35.648362 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:35.648381 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:35.648750 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:35.649066 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:35.649494 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:35.650512 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:35.651415 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:35.651949 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:35.652098 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:35.653003 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:35.653008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:35.653022 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:35.653445 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:35.653511 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:35.653536 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:35.653684 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:35.653872 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 23:05:35.654021 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":154,"threads_started":1}
I20260810 23:05:35.655267 15748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b: No bootstrap required, opened a new log
I20260810 23:05:35.658069 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:35.659468 15748 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbd69ceedb84a5188e6e98b52f2885b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } }
I20260810 23:05:35.659773 15748 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:35.659842 15748 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbd69ceedb84a5188e6e98b52f2885b, State: Initialized, Role: FOLLOWER
I20260810 23:05:35.660323 15748 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbd69ceedb84a5188e6e98b52f2885b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } }
I20260810 23:05:35.660483 15748 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:35.660547 15748 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:35.660645 15748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:35.661876 15748 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbd69ceedb84a5188e6e98b52f2885b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } }
I20260810 23:05:35.662288 15748 leader_election.cc:302] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbbd69ceedb84a5188e6e98b52f2885b; no voters:
I20260810 23:05:35.662640 15748 leader_election.cc:288] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:35.662758 15767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:35.662992 15767 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [term 1 LEADER]: Becoming Leader. State: Replica: cbbd69ceedb84a5188e6e98b52f2885b, State: Running, Role: LEADER
I20260810 23:05:35.663312 15767 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbd69ceedb84a5188e6e98b52f2885b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } }
I20260810 23:05:35.664201 15748 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:35.664619 15767 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbbd69ceedb84a5188e6e98b52f2885b. Latest consensus state: current_term: 1 leader_uuid: "cbbd69ceedb84a5188e6e98b52f2885b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbd69ceedb84a5188e6e98b52f2885b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } } }
I20260810 23:05:35.664776 15767 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:35.664619 15771 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbbd69ceedb84a5188e6e98b52f2885b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbd69ceedb84a5188e6e98b52f2885b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } } }
I20260810 23:05:35.664860 15771 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:35.665251 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:35.660399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39595 (local address 127.0.0.1:35113)
0810 23:05:35.660840 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:35.660845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:35.660861 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:05:35.660900 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:35.660903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:35.660948 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:35.661039 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:35.661271 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:35.662008 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:35.663018 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:35.663361 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:35.663464 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:35.664582 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:35.664586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:35.664589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:35.664604 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:35.664633 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:35.664647 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:35.664751 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:35.665081 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 23:05:35.665115 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":356}
I20260810 23:05:35.665866 15774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:35.669745 15774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:35.675093 15774 catalog_manager.cc:1347] Generated new cluster ID: c71c1069090a48a4b9ee74816698aeac
I20260810 23:05:35.675165 15774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:35.684832 15774 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:35.686077 15774 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:35.694159 15774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbbd69ceedb84a5188e6e98b52f2885b: Generated new TSK 0
I20260810 23:05:35.695137 15774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:35.702565 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 15634
W20260810 23:05:35.713923 15614 connection.cc:535] client connection to 127.0.0.1:35113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35113): Connection reset by peer (error 104)
2026-08-10T23:05:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1569 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:35 dist-test-slave-6bbx krb5kdc[15797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:35 dist-test-slave-6bbx krb5kdc[15797](info): set up 2 sockets
Aug 10 23:05:35 dist-test-slave-6bbx krb5kdc[15797](info): commencing operation
krb5kdc: starting...
W20260810 23:05:36.846760 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:36 dist-test-slave-6bbx krb5kdc[15797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:36.984861 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:37.104650 15814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:37.104959 15814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:37.105022 15814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:37.109088 15814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:37.109165 15814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:37.109192 15814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:37.109213 15814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:37.109241 15814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:37.114904 15814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:37.116138 15814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:37.117239 15814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:37.126696 15822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:37.126724 15820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:37.126714 15814 server_base.cc:1034] running on GCE node
W20260810 23:05:37.126739 15824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:37.127578 15814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:37.128750 15814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:37.129920 15814 hybrid_clock.cc:648] HybridClock initialized: now 1786403137129895 us; error 36 us; skew 500 ppm
Aug 10 23:05:37 dist-test-slave-6bbx krb5kdc[15797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:37.132769 15814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:37.133684 15814 webserver.cc:460] Webserver started at http://127.0.0.1:33089/ using document root <none> and password file <none>
I20260810 23:05:37.134248 15814 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:37.134312 15814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:37.134526 15814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:37.136649 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "b3c29a645dd14df0a7f39a839c9a446c"
format_stamp: "Formatted at 2026-08-10 23:05:37 on dist-test-slave-6bbx"
server_key: "0492e16fb12bf81150eb348410b9eb6b"
server_key_iv: "1a08507b16dade5c8d49d7cd95505f5a"
server_key_version: "encryptionkey@0"
I20260810 23:05:37.137178 15814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "b3c29a645dd14df0a7f39a839c9a446c"
format_stamp: "Formatted at 2026-08-10 23:05:37 on dist-test-slave-6bbx"
server_key: "0492e16fb12bf81150eb348410b9eb6b"
server_key_iv: "1a08507b16dade5c8d49d7cd95505f5a"
server_key_version: "encryptionkey@0"
I20260810 23:05:37.140928 15814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:05:37.143386 15836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:37.144804 15814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:05:37.144951 15814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "b3c29a645dd14df0a7f39a839c9a446c"
format_stamp: "Formatted at 2026-08-10 23:05:37 on dist-test-slave-6bbx"
server_key: "0492e16fb12bf81150eb348410b9eb6b"
server_key_iv: "1a08507b16dade5c8d49d7cd95505f5a"
server_key_version: "encryptionkey@0"
I20260810 23:05:37.145059 15814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:37.149798 15814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:37.152933 15814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:37.153101 15814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:37.159616 15814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38823
I20260810 23:05:37.159679 15924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38823 every 8 connection(s)
I20260810 23:05:37.160547 15814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:37.163117 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 15814
I20260810 23:05:37.163278 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:37.163497 15925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:37.163630 15584 external_mini_cluster.cc:1428] Setting key 2eb8cb459b01d23b7ac11eae3a93c141
I20260810 23:05:37.168919 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:37.164972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41465 (local address 127.0.0.1:38823)
0810 23:05:37.165366 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:37.165382 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:37.165418 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:05:37.166168 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:37.166188 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:37.166479 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:37.166772 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:37.167624 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:37.167629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:37.167642 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:37.167653 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:37.167752 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:37.167778 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:37.167895 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:37.168001 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 23:05:37.168148 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260810 23:05:37.170614 15925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c: Bootstrap starting.
I20260810 23:05:37.171924 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:37.173538 15925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:37.174458 15925 log.cc:824] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:37.176179 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:37.174356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57331 (local address 127.0.0.1:38823)
0810 23:05:37.174484 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:37.174489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:37.174674 (+ 185us) server_negotiation.cc:407] Connection header received
0810 23:05:37.174780 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:37.174783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:37.174824 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:37.174911 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:37.175720 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:37.175724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:37.175727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:37.175741 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:37.175773 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:37.175786 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:37.175880 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:37.176006 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 23:05:37.176046 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:05:37.177166 15925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c: No bootstrap required, opened a new log
I20260810 23:05:37.181130 15925 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c29a645dd14df0a7f39a839c9a446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38823 } }
I20260810 23:05:37.181454 15925 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:37.181525 15925 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c29a645dd14df0a7f39a839c9a446c, State: Initialized, Role: FOLLOWER
I20260810 23:05:37.182078 15925 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c29a645dd14df0a7f39a839c9a446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38823 } }
I20260810 23:05:37.182322 15925 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:37.182411 15925 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:37.182533 15925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:37.183598 15925 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c29a645dd14df0a7f39a839c9a446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38823 } }
I20260810 23:05:37.184031 15925 leader_election.cc:302] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3c29a645dd14df0a7f39a839c9a446c; no voters:
I20260810 23:05:37.184419 15925 leader_election.cc:288] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:37.184552 15939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:37.184813 15939 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [term 1 LEADER]: Becoming Leader. State: Replica: b3c29a645dd14df0a7f39a839c9a446c, State: Running, Role: LEADER
I20260810 23:05:37.185220 15939 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c29a645dd14df0a7f39a839c9a446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38823 } }
I20260810 23:05:37.185930 15925 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:37.187047 15941 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3c29a645dd14df0a7f39a839c9a446c. Latest consensus state: current_term: 1 leader_uuid: "b3c29a645dd14df0a7f39a839c9a446c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c29a645dd14df0a7f39a839c9a446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38823 } } }
I20260810 23:05:37.187201 15941 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:37.187497 15940 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3c29a645dd14df0a7f39a839c9a446c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c29a645dd14df0a7f39a839c9a446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38823 } } }
I20260810 23:05:37.187588 15940 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:37.187610 15950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:37.190554 15950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:37.196887 15950 catalog_manager.cc:1347] Generated new cluster ID: 1688d26575654c3d8a896570f5f8c68e
I20260810 23:05:37.196981 15950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:37.213054 15950 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:37.213440 15950 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:37.229455 15950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3c29a645dd14df0a7f39a839c9a446c: Generated new TSK 0
I20260810 23:05:37.230470 15950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:37.242707 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 15814
2026-08-10T23:05:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1543 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:37 dist-test-slave-6bbx krb5kdc[15966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:37 dist-test-slave-6bbx krb5kdc[15966](info): set up 2 sockets
Aug 10 23:05:37 dist-test-slave-6bbx krb5kdc[15966](info): commencing operation
krb5kdc: starting...
W20260810 23:05:38.399178 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:38 dist-test-slave-6bbx krb5kdc[15966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:38.529098 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46287
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:38.644963 15994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:38.645260 15994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:38.645323 15994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:38.649317 15994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:38.649382 15994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:38.649410 15994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:38.649431 15994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:38.649461 15994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:38.654873 15994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:38.656090 15994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:38.657151 15994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:38.665674 16001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:38.665781 16003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:38.665704 16007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:38.665903 15994 server_base.cc:1034] running on GCE node
I20260810 23:05:38.666477 15994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:38.667518 15994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:38.668670 15994 hybrid_clock.cc:648] HybridClock initialized: now 1786403138668655 us; error 28 us; skew 500 ppm
Aug 10 23:05:38 dist-test-slave-6bbx krb5kdc[15966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:38.671365 15994 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:38.672295 15994 webserver.cc:460] Webserver started at http://127.0.0.1:32865/ using document root <none> and password file <none>
I20260810 23:05:38.672878 15994 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:38.672940 15994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:38.673126 15994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:38.675212 15994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "fd3c0905c1fb44e9adfd776350be3fb4"
format_stamp: "Formatted at 2026-08-10 23:05:38 on dist-test-slave-6bbx"
server_key: "5ab3e93af01af5dda183be66edb9a5a1"
server_key_iv: "b1625d6776626d360538fcc1dea133d8"
server_key_version: "encryptionkey@0"
I20260810 23:05:38.675782 15994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "fd3c0905c1fb44e9adfd776350be3fb4"
format_stamp: "Formatted at 2026-08-10 23:05:38 on dist-test-slave-6bbx"
server_key: "5ab3e93af01af5dda183be66edb9a5a1"
server_key_iv: "b1625d6776626d360538fcc1dea133d8"
server_key_version: "encryptionkey@0"
I20260810 23:05:38.679786 15994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:38.682551 16017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:38.684131 15994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:38.684266 15994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "fd3c0905c1fb44e9adfd776350be3fb4"
format_stamp: "Formatted at 2026-08-10 23:05:38 on dist-test-slave-6bbx"
server_key: "5ab3e93af01af5dda183be66edb9a5a1"
server_key_iv: "b1625d6776626d360538fcc1dea133d8"
server_key_version: "encryptionkey@0"
I20260810 23:05:38.684399 15994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:38.704808 15994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:38.708180 15994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:38.708380 15994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:38.715587 15994 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46287
I20260810 23:05:38.715616 16094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46287 every 8 connection(s)
I20260810 23:05:38.716706 15994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:38.719831 16097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:38.725559 16097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4: Bootstrap starting.
I20260810 23:05:38.726867 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 15994
I20260810 23:05:38.726996 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:38.727281 15584 external_mini_cluster.cc:1428] Setting key 7099c310da30dff78ba9944cc7938f8b
I20260810 23:05:38.728354 16097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:38.729205 16097 log.cc:824] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:38.731427 16097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4: No bootstrap required, opened a new log
I20260810 23:05:38.735584 16097 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3c0905c1fb44e9adfd776350be3fb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46287 } }
I20260810 23:05:38.735884 16097 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:38.735942 16097 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3c0905c1fb44e9adfd776350be3fb4, State: Initialized, Role: FOLLOWER
I20260810 23:05:38.736403 16097 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3c0905c1fb44e9adfd776350be3fb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46287 } }
I20260810 23:05:38.736541 16097 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:38.736613 16097 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:38.736716 16097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 23:05:38 dist-test-slave-6bbx krb5kdc[15966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:05:38.737859 16097 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3c0905c1fb44e9adfd776350be3fb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46287 } }
I20260810 23:05:38.738225 16097 leader_election.cc:302] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd3c0905c1fb44e9adfd776350be3fb4; no voters:
I20260810 23:05:38.738582 16097 leader_election.cc:288] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:38.738651 16102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:38.738865 16102 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [term 1 LEADER]: Becoming Leader. State: Replica: fd3c0905c1fb44e9adfd776350be3fb4, State: Running, Role: LEADER
I20260810 23:05:38.739248 16102 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3c0905c1fb44e9adfd776350be3fb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46287 } }
I20260810 23:05:38.740029 16097 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:38.741015 16105 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd3c0905c1fb44e9adfd776350be3fb4. Latest consensus state: current_term: 1 leader_uuid: "fd3c0905c1fb44e9adfd776350be3fb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3c0905c1fb44e9adfd776350be3fb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46287 } } }
I20260810 23:05:38.741146 16105 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:38.741374 16105 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd3c0905c1fb44e9adfd776350be3fb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd3c0905c1fb44e9adfd776350be3fb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46287 } } }
I20260810 23:05:38.741448 16105 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:38.742014 16121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:38.743152 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:38.728466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34689 (local address 127.0.0.1:46287)
0810 23:05:38.728950 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:38.728970 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:38.729016 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:05:38.729839 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:38.729859 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:38.730280 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:38.730645 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:38.731342 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:38.732813 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:38.733682 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:38.734428 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:38.734595 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:38.737470 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:38.737476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:38.737492 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:38.737522 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:38.739427 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:38.739923 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:38.739928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:38.739933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:38.740053 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:38.740336 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:38.740339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:38.740341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:38.740904 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:38.741759 (+ 855us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:38.741925 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:05:38.742256 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":138,"threads_started":1}
I20260810 23:05:38.745412 16121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:38.746723 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:38.752223 16121 catalog_manager.cc:1347] Generated new cluster ID: c095888bd5df4278a6a95b65c962a22d
I20260810 23:05:38.752415 16121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:38.757741 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:38.749906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52201 (local address 127.0.0.1:46287)
0810 23:05:38.750049 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:38.750054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:38.750072 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:05:38.750117 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:38.750120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:38.750170 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:38.750257 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:38.751290 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:38.752059 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:38.753029 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:38.753282 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:38.753374 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:38.754786 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:38.754789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:38.754792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:38.754817 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:38.756162 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:38.756579 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:38.756583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:38.756584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:38.756663 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:38.756913 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:38.756917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:38.756919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:38.757099 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:38.757219 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:38.757460 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 23:05:38.757575 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:05:38.764284 16121 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:38.765175 16121 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:38.777436 16121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd3c0905c1fb44e9adfd776350be3fb4: Generated new TSK 0
I20260810 23:05:38.778192 16121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:38.786196 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 15994
2026-08-10T23:05:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1547 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:38 dist-test-slave-6bbx krb5kdc[16148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:38 dist-test-slave-6bbx krb5kdc[16148](info): set up 2 sockets
Aug 10 23:05:38 dist-test-slave-6bbx krb5kdc[16148](info): commencing operation
krb5kdc: starting...
W20260810 23:05:39.961163 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:40 dist-test-slave-6bbx krb5kdc[16148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:40.088881 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:40.222090 16171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:40.222425 16171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:40.222488 16171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:40.226573 16171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:40.226644 16171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:40.226670 16171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:40.226692 16171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:40.226722 16171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:40.232132 16171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:40.233201 16171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:40.234254 16171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:40.242970 16178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:40.242967 16177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:40.242966 16181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:40.243323 16171 server_base.cc:1034] running on GCE node
I20260810 23:05:40.243947 16171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:40.245064 16171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:40.246212 16171 hybrid_clock.cc:648] HybridClock initialized: now 1786403140246185 us; error 39 us; skew 500 ppm
Aug 10 23:05:40 dist-test-slave-6bbx krb5kdc[16148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:40.248906 16171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:40.249821 16171 webserver.cc:460] Webserver started at http://127.0.0.1:33031/ using document root <none> and password file <none>
I20260810 23:05:40.250372 16171 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:40.250440 16171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:40.250650 16171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:40.252781 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "c8309f89efe2474d97f11195bab839a8"
format_stamp: "Formatted at 2026-08-10 23:05:40 on dist-test-slave-6bbx"
server_key: "c67daf1efe562119205ee9e25b58d22f"
server_key_iv: "657eb3e97b61448c28579cd8a3843dd4"
server_key_version: "encryptionkey@0"
I20260810 23:05:40.253333 16171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "c8309f89efe2474d97f11195bab839a8"
format_stamp: "Formatted at 2026-08-10 23:05:40 on dist-test-slave-6bbx"
server_key: "c67daf1efe562119205ee9e25b58d22f"
server_key_iv: "657eb3e97b61448c28579cd8a3843dd4"
server_key_version: "encryptionkey@0"
I20260810 23:05:40.257117 16171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:40.259501 16195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:40.260737 16171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:05:40.260882 16171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "c8309f89efe2474d97f11195bab839a8"
format_stamp: "Formatted at 2026-08-10 23:05:40 on dist-test-slave-6bbx"
server_key: "c67daf1efe562119205ee9e25b58d22f"
server_key_iv: "657eb3e97b61448c28579cd8a3843dd4"
server_key_version: "encryptionkey@0"
I20260810 23:05:40.260980 16171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:40.279539 16171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:40.282891 16171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:40.283087 16171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:40.290591 16171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38789
I20260810 23:05:40.290614 16281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38789 every 8 connection(s)
I20260810 23:05:40.291728 16171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:40.294754 16282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:40.299013 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 16171
I20260810 23:05:40.299165 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:40.299431 15584 external_mini_cluster.cc:1428] Setting key ec578534d47c0b330a74c3c87172f805
I20260810 23:05:40.300662 16282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8: Bootstrap starting.
I20260810 23:05:40.303017 16282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:40.303774 16282 log.cc:824] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:40.305641 16282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8: No bootstrap required, opened a new log
I20260810 23:05:40.308516 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:40.300931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34147 (local address 127.0.0.1:38789)
0810 23:05:40.301392 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:40.301408 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:40.301447 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:05:40.302096 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:40.302128 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:40.302509 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:40.302835 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:40.303279 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.304323 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.305254 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.305725 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.305894 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:40.306724 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:40.306730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:40.306744 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:40.307127 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:40.307196 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:40.307219 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:40.307310 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:40.307451 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 23:05:40.307653 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":136,"threads_started":1}
I20260810 23:05:40.309297 16282 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8309f89efe2474d97f11195bab839a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38789 } }
I20260810 23:05:40.309556 16282 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:40.309628 16282 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8309f89efe2474d97f11195bab839a8, State: Initialized, Role: FOLLOWER
I20260810 23:05:40.310125 16282 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8309f89efe2474d97f11195bab839a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38789 } }
I20260810 23:05:40.310292 16282 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:40.310362 16282 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:40.310463 16282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:40.311419 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:40.311579 16282 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8309f89efe2474d97f11195bab839a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38789 } }
I20260810 23:05:40.311921 16282 leader_election.cc:302] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8309f89efe2474d97f11195bab839a8; no voters:
I20260810 23:05:40.312250 16282 leader_election.cc:288] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:40.312335 16292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:40.312502 16292 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [term 1 LEADER]: Becoming Leader. State: Replica: c8309f89efe2474d97f11195bab839a8, State: Running, Role: LEADER
I20260810 23:05:40.312877 16292 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8309f89efe2474d97f11195bab839a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38789 } }
I20260810 23:05:40.313560 16282 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:40.314556 16294 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8309f89efe2474d97f11195bab839a8. Latest consensus state: current_term: 1 leader_uuid: "c8309f89efe2474d97f11195bab839a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8309f89efe2474d97f11195bab839a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38789 } } }
I20260810 23:05:40.314702 16294 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:40.315017 16293 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8309f89efe2474d97f11195bab839a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8309f89efe2474d97f11195bab839a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38789 } } }
I20260810 23:05:40.315110 16293 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:40.315452 16312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:40.318096 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:40.313524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57507 (local address 127.0.0.1:38789)
0810 23:05:40.313626 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:40.313630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:40.313898 (+ 268us) server_negotiation.cc:407] Connection header received
0810 23:05:40.314039 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:40.314043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:40.314090 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:40.314139 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:40.314562 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.315323 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.316416 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.316652 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.316741 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:40.317582 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:40.317586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:40.317589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:40.317602 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:40.317633 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:40.317647 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:40.317704 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:40.317894 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 23:05:40.317951 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 23:05:40.318686 16312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:40.324430 16312 catalog_manager.cc:1347] Generated new cluster ID: 87c62ed8c9b9483d89244b1368834847
I20260810 23:05:40.324489 16312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:40.334290 16312 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:40.335527 16312 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:40.344569 16312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8309f89efe2474d97f11195bab839a8: Generated new TSK 0
I20260810 23:05:40.345330 16312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:40.354827 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 16171
2026-08-10T23:05:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1561 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:40 dist-test-slave-6bbx krb5kdc[16329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:40 dist-test-slave-6bbx krb5kdc[16329](info): set up 2 sockets
Aug 10 23:05:40 dist-test-slave-6bbx krb5kdc[16329](info): commencing operation
krb5kdc: starting...
W20260810 23:05:41.511984 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:41 dist-test-slave-6bbx krb5kdc[16329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:41.642446 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:41.760468 16347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:41.760761 16347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:41.760804 16347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:41.764896 16347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:41.764961 16347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:41.764981 16347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:41.764997 16347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:41.765013 16347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:41.770478 16347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:41.771643 16347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:41.772763 16347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:41.781600 16358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:41.781589 16360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:41.781589 16356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:41.781715 16347 server_base.cc:1034] running on GCE node
I20260810 23:05:41.782554 16347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:41.783694 16347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:41.784907 16347 hybrid_clock.cc:648] HybridClock initialized: now 1786403141784894 us; error 26 us; skew 500 ppm
Aug 10 23:05:41 dist-test-slave-6bbx krb5kdc[16329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:41.792007 16347 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:41.793042 16347 webserver.cc:460] Webserver started at http://127.0.0.1:38825/ using document root <none> and password file <none>
I20260810 23:05:41.793649 16347 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:41.793723 16347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:41.793956 16347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:41.796267 16347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "f422ce8df98742f186a180fa74daa8ad"
format_stamp: "Formatted at 2026-08-10 23:05:41 on dist-test-slave-6bbx"
server_key: "3afc34d9f201257ffaceb48cbf927fe3"
server_key_iv: "0ea359015c6edab2c0f4d7e14ee496a1"
server_key_version: "encryptionkey@0"
I20260810 23:05:41.796839 16347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "f422ce8df98742f186a180fa74daa8ad"
format_stamp: "Formatted at 2026-08-10 23:05:41 on dist-test-slave-6bbx"
server_key: "3afc34d9f201257ffaceb48cbf927fe3"
server_key_iv: "0ea359015c6edab2c0f4d7e14ee496a1"
server_key_version: "encryptionkey@0"
I20260810 23:05:41.800711 16347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:41.803390 16377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.804755 16347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:41.804908 16347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "f422ce8df98742f186a180fa74daa8ad"
format_stamp: "Formatted at 2026-08-10 23:05:41 on dist-test-slave-6bbx"
server_key: "3afc34d9f201257ffaceb48cbf927fe3"
server_key_iv: "0ea359015c6edab2c0f4d7e14ee496a1"
server_key_version: "encryptionkey@0"
I20260810 23:05:41.805017 16347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:41.809939 16347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:41.813516 16347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:41.813724 16347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:41.820865 16347 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32991
I20260810 23:05:41.820885 16458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32991 every 8 connection(s)
I20260810 23:05:41.821906 16347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:41.824950 16459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:41.830322 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 16347
I20260810 23:05:41.830492 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:41.830791 15584 external_mini_cluster.cc:1428] Setting key 10d61ef3d82b0f55d0e49ea695b855c9
I20260810 23:05:41.831498 16459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad: Bootstrap starting.
I20260810 23:05:41.834035 16459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:41.834959 16459 log.cc:824] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:41.836544 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:41.832317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48709 (local address 127.0.0.1:32991)
0810 23:05:41.832767 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:41.832782 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:41.832820 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:05:41.833478 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:41.833497 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:41.833808 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:41.834097 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:41.834976 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:41.834982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:41.834995 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:41.835007 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:41.835088 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:41.835121 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:41.835226 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:41.835481 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 23:05:41.835717 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260810 23:05:41.837227 16459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad: No bootstrap required, opened a new log
I20260810 23:05:41.839684 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:41.841828 16459 raft_consensus.cc:348] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422ce8df98742f186a180fa74daa8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } }
I20260810 23:05:41.842187 16459 raft_consensus.cc:374] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:41.842257 16459 raft_consensus.cc:729] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f422ce8df98742f186a180fa74daa8ad, State: Initialized, Role: FOLLOWER
I20260810 23:05:41.842758 16459 consensus_queue.cc:260] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422ce8df98742f186a180fa74daa8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } }
I20260810 23:05:41.842939 16459 raft_consensus.cc:388] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:41.843014 16459 raft_consensus.cc:482] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:41.843122 16459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:41.844055 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:41.842213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40621 (local address 127.0.0.1:32991)
0810 23:05:41.842350 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:41.842355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:41.842634 (+ 279us) server_negotiation.cc:407] Connection header received
0810 23:05:41.842737 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:41.842740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:41.842783 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:41.842884 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:41.843650 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:41.843654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:41.843690 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:41.843704 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:41.843734 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:41.843747 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:41.843843 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:41.843896 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 23:05:41.843935 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:05:41.844383 16459 raft_consensus.cc:504] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422ce8df98742f186a180fa74daa8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } }
I20260810 23:05:41.844769 16459 leader_election.cc:302] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f422ce8df98742f186a180fa74daa8ad; no voters:
I20260810 23:05:41.845144 16459 leader_election.cc:288] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:41.845225 16473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:41.845436 16473 raft_consensus.cc:686] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [term 1 LEADER]: Becoming Leader. State: Replica: f422ce8df98742f186a180fa74daa8ad, State: Running, Role: LEADER
I20260810 23:05:41.845805 16473 consensus_queue.cc:237] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422ce8df98742f186a180fa74daa8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } }
I20260810 23:05:41.846503 16459 sys_catalog.cc:564] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:41.847465 16475 sys_catalog.cc:455] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader f422ce8df98742f186a180fa74daa8ad. Latest consensus state: current_term: 1 leader_uuid: "f422ce8df98742f186a180fa74daa8ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422ce8df98742f186a180fa74daa8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } } }
I20260810 23:05:41.847594 16475 sys_catalog.cc:458] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:41.847862 16474 sys_catalog.cc:455] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f422ce8df98742f186a180fa74daa8ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f422ce8df98742f186a180fa74daa8ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } } }
I20260810 23:05:41.847966 16474 sys_catalog.cc:458] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad [sys.catalog]: This master's current role is: LEADER
W20260810 23:05:41.851534 16498 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:05:41.851610 16498 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:41.851756 16487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:41.852663 16487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:41.858759 16487 catalog_manager.cc:1347] Generated new cluster ID: b2f48a4538564419ad2514b9c9c94066
I20260810 23:05:41.858862 16487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:41.872496 16487 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:41.872861 16487 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:41.879374 16487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f422ce8df98742f186a180fa74daa8ad: Generated new TSK 0
I20260810 23:05:41.880172 16487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:41.889393 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 16347
2026-08-10T23:05:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1543 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:41 dist-test-slave-6bbx krb5kdc[16507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:41 dist-test-slave-6bbx krb5kdc[16507](info): set up 2 sockets
Aug 10 23:05:41 dist-test-slave-6bbx krb5kdc[16507](info): commencing operation
krb5kdc: starting...
W20260810 23:05:43.064872 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:43 dist-test-slave-6bbx krb5kdc[16507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:43.206472 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42039
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:43.329994 16529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:43.330359 16529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:43.330406 16529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:43.334724 16529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:43.334811 16529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:43.334830 16529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:43.334846 16529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:43.334860 16529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:43.340607 16529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:43.341842 16529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:43.342958 16529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:43.352560 16537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:43.352607 16534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:43.352635 16539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:43.352842 16529 server_base.cc:1034] running on GCE node
I20260810 23:05:43.353569 16529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:43.354852 16529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:43.356055 16529 hybrid_clock.cc:648] HybridClock initialized: now 1786403143356022 us; error 45 us; skew 500 ppm
Aug 10 23:05:43 dist-test-slave-6bbx krb5kdc[16507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:43.359134 16529 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:43.360248 16529 webserver.cc:460] Webserver started at http://127.0.0.1:38607/ using document root <none> and password file <none>
I20260810 23:05:43.360855 16529 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:43.360927 16529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:43.361140 16529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:43.363394 16529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "657f4add533842b99d0bb02f7291169f"
format_stamp: "Formatted at 2026-08-10 23:05:43 on dist-test-slave-6bbx"
server_key: "c8ef9c6b1b0c01f1f40cea6718fc6db6"
server_key_iv: "e779706fb449496d2b843b07c5f1d9ad"
server_key_version: "encryptionkey@0"
I20260810 23:05:43.364030 16529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "657f4add533842b99d0bb02f7291169f"
format_stamp: "Formatted at 2026-08-10 23:05:43 on dist-test-slave-6bbx"
server_key: "c8ef9c6b1b0c01f1f40cea6718fc6db6"
server_key_iv: "e779706fb449496d2b843b07c5f1d9ad"
server_key_version: "encryptionkey@0"
I20260810 23:05:43.368248 16529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:43.371076 16553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:43.373284 16529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:43.373420 16529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "657f4add533842b99d0bb02f7291169f"
format_stamp: "Formatted at 2026-08-10 23:05:43 on dist-test-slave-6bbx"
server_key: "c8ef9c6b1b0c01f1f40cea6718fc6db6"
server_key_iv: "e779706fb449496d2b843b07c5f1d9ad"
server_key_version: "encryptionkey@0"
I20260810 23:05:43.373525 16529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:43.392537 16529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:43.396106 16529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:43.396330 16529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:43.404184 16529 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42039
I20260810 23:05:43.404264 16636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42039 every 8 connection(s)
I20260810 23:05:43.405287 16529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:43.408929 16637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:43.415549 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 16529
I20260810 23:05:43.415568 16637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f: Bootstrap starting.
I20260810 23:05:43.415779 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:43.416105 15584 external_mini_cluster.cc:1428] Setting key e2c5b64131262bdbde26c04d32d6479c
I20260810 23:05:43.418740 16637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:43.419569 16637 log.cc:824] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:43.421954 16637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f: No bootstrap required, opened a new log
Aug 10 23:05:43 dist-test-slave-6bbx krb5kdc[16507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:05:43.426671 16637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657f4add533842b99d0bb02f7291169f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } }
I20260810 23:05:43.427003 16637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:43.427071 16637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 657f4add533842b99d0bb02f7291169f, State: Initialized, Role: FOLLOWER
I20260810 23:05:43.427594 16637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657f4add533842b99d0bb02f7291169f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } }
I20260810 23:05:43.427790 16637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:43.427872 16637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:43.427989 16637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:43.429320 16637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657f4add533842b99d0bb02f7291169f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } }
I20260810 23:05:43.429780 16637 leader_election.cc:302] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 657f4add533842b99d0bb02f7291169f; no voters:
I20260810 23:05:43.430220 16637 leader_election.cc:288] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:43.430325 16644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:43.430547 16644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [term 1 LEADER]: Becoming Leader. State: Replica: 657f4add533842b99d0bb02f7291169f, State: Running, Role: LEADER
I20260810 23:05:43.430979 16644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657f4add533842b99d0bb02f7291169f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } }
I20260810 23:05:43.431879 16637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:43.433043 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.417697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:44643 (local address 127.0.0.1:42039)
0810 23:05:43.418187 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:43.418204 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:43.418243 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:05:43.418986 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:43.419010 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:43.419413 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:43.419791 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:43.420720 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.421774 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.422970 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.423533 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.423699 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.426868 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:43.426874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:43.426891 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:43.426929 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:43.428870 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:43.429622 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:43.429629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:43.429637 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:43.429824 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:43.430328 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:43.430332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:43.430334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:43.430979 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:43.431232 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:43.431574 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 23:05:43.431966 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":146,"threads_started":1}
I20260810 23:05:43.433071 16646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 657f4add533842b99d0bb02f7291169f. Latest consensus state: current_term: 1 leader_uuid: "657f4add533842b99d0bb02f7291169f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657f4add533842b99d0bb02f7291169f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } } }
I20260810 23:05:43.433192 16646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:43.434343 16647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "657f4add533842b99d0bb02f7291169f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "657f4add533842b99d0bb02f7291169f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42039 } } }
I20260810 23:05:43.435259 16647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:43.436780 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 23:05:43.437691 16667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:05:43.437881 16667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:43.437997 16658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:43.439111 16658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:43.446617 16658 catalog_manager.cc:1347] Generated new cluster ID: 4052c5e3d4f84145ae4d6d545b1fb929
I20260810 23:05:43.446702 16658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:43.448504 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.439237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:43193 (local address 127.0.0.1:42039)
0810 23:05:43.439381 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:43.439386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:43.439803 (+ 417us) server_negotiation.cc:407] Connection header received
0810 23:05:43.439944 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:43.439947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:43.440003 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:43.440095 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:43.440994 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.441604 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.442502 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.442774 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.442870 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.444609 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:43.444618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:43.444621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:43.444655 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:43.446438 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:43.447066 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:43.447069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:43.447070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:43.447163 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:43.447510 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:43.447514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:43.447516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:43.447792 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:43.447943 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:43.448242 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 23:05:43.448343 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 23:05:43.466656 16658 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:43.467801 16658 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:43.473893 16658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 657f4add533842b99d0bb02f7291169f: Generated new TSK 0
I20260810 23:05:43.474730 16658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:43.484887 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 16529
W20260810 23:05:43.499993 16505 connection.cc:535] client connection to 127.0.0.1:42039 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42039): Connection reset by peer (error 104)
2026-08-10T23:05:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1590 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:43 dist-test-slave-6bbx krb5kdc[16686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:43 dist-test-slave-6bbx krb5kdc[16686](info): set up 2 sockets
Aug 10 23:05:43 dist-test-slave-6bbx krb5kdc[16686](info): commencing operation
krb5kdc: starting...
W20260810 23:05:44.663365 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:44 dist-test-slave-6bbx krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:44.804162 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:44.926484 16706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:44.926800 16706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:44.926865 16706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:44.931077 16706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:44.931173 16706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:44.931200 16706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:44.931223 16706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:44.931253 16706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:44.936924 16706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:44.938097 16706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:44.939186 16706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:44.948554 16719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:44.948679 16706 server_base.cc:1034] running on GCE node
W20260810 23:05:44.948559 16716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:44.948559 16726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:44.949626 16706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:44.950888 16706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:44.952099 16706 hybrid_clock.cc:648] HybridClock initialized: now 1786403144952061 us; error 50 us; skew 500 ppm
Aug 10 23:05:44 dist-test-slave-6bbx krb5kdc[16686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:44.955206 16706 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:44.956266 16706 webserver.cc:460] Webserver started at http://127.0.0.1:39471/ using document root <none> and password file <none>
I20260810 23:05:44.956851 16706 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:44.956923 16706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:44.957136 16706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:44.959466 16706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "47701d4d79e2464798024ea4a40e0377"
format_stamp: "Formatted at 2026-08-10 23:05:44 on dist-test-slave-6bbx"
server_key: "2ccda13dc937b67b628a0d89a1e12499"
server_key_iv: "7faf457049f3e91499442006471310e2"
server_key_version: "encryptionkey@0"
I20260810 23:05:44.960161 16706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "47701d4d79e2464798024ea4a40e0377"
format_stamp: "Formatted at 2026-08-10 23:05:44 on dist-test-slave-6bbx"
server_key: "2ccda13dc937b67b628a0d89a1e12499"
server_key_iv: "7faf457049f3e91499442006471310e2"
server_key_version: "encryptionkey@0"
I20260810 23:05:44.964520 16706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:44.967849 16738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:44.969350 16706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:05:44.969489 16706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "47701d4d79e2464798024ea4a40e0377"
format_stamp: "Formatted at 2026-08-10 23:05:44 on dist-test-slave-6bbx"
server_key: "2ccda13dc937b67b628a0d89a1e12499"
server_key_iv: "7faf457049f3e91499442006471310e2"
server_key_version: "encryptionkey@0"
I20260810 23:05:44.969609 16706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:44.981094 16706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:44.984486 16706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:44.984663 16706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:44.992473 16706 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43489
I20260810 23:05:44.992515 16810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43489 every 8 connection(s)
I20260810 23:05:44.993587 16706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:44.996924 16811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:45.002687 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 16706
I20260810 23:05:45.002851 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:45.003125 15584 external_mini_cluster.cc:1428] Setting key 06e78b17e31d9c5148a027a38bcb0eb3
I20260810 23:05:45.003551 16811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377: Bootstrap starting.
I20260810 23:05:45.006170 16811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:45.007099 16811 log.cc:824] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:45.009609 16811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377: No bootstrap required, opened a new log
I20260810 23:05:45.013262 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.004710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:36937 (local address 127.0.0.1:43489)
0810 23:05:45.005170 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.005188 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.005229 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:05:45.005935 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.005954 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.006364 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.006669 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.007065 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.008349 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.009595 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.010128 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.010253 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.011196 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.011201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.011216 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:45.011706 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.011774 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:45.011805 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.011911 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.012144 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.012337 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":119,"threads_started":1}
I20260810 23:05:45.013860 16811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47701d4d79e2464798024ea4a40e0377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } }
I20260810 23:05:45.014111 16811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:45.014173 16811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47701d4d79e2464798024ea4a40e0377, State: Initialized, Role: FOLLOWER
I20260810 23:05:45.014698 16811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47701d4d79e2464798024ea4a40e0377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } }
I20260810 23:05:45.014871 16811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:45.014936 16811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:45.015038 16811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:45.016201 16811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47701d4d79e2464798024ea4a40e0377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } }
I20260810 23:05:45.016654 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:45.016592 16811 leader_election.cc:302] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47701d4d79e2464798024ea4a40e0377; no voters:
I20260810 23:05:45.016968 16811 leader_election.cc:288] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:45.017072 16816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:45.017318 16816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [term 1 LEADER]: Becoming Leader. State: Replica: 47701d4d79e2464798024ea4a40e0377, State: Running, Role: LEADER
I20260810 23:05:45.017735 16816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47701d4d79e2464798024ea4a40e0377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } }
I20260810 23:05:45.018447 16811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:45.019436 16818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47701d4d79e2464798024ea4a40e0377. Latest consensus state: current_term: 1 leader_uuid: "47701d4d79e2464798024ea4a40e0377" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47701d4d79e2464798024ea4a40e0377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } } }
I20260810 23:05:45.019594 16818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:45.019961 16817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47701d4d79e2464798024ea4a40e0377" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47701d4d79e2464798024ea4a40e0377" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43489 } } }
I20260810 23:05:45.020051 16817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377 [sys.catalog]: This master's current role is: LEADER
W20260810 23:05:45.023307 16846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:05:45.023402 16846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:45.023502 16837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:45.024252 16837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:45.024470 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.019313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:42899 (local address 127.0.0.1:43489)
0810 23:05:45.019426 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.019431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.019708 (+ 277us) server_negotiation.cc:407] Connection header received
0810 23:05:45.019928 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.019932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.019986 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.020038 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.020490 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.021165 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.022295 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.022617 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.022693 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.023908 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.023912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.023915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:45.023929 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.023960 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:45.023973 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.024031 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.024283 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.024327 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:05:45.030334 16837 catalog_manager.cc:1347] Generated new cluster ID: d46315cdbec5455aad39ad4649d21035
I20260810 23:05:45.030429 16837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:45.050794 16837 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:45.052330 16837 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:45.070686 16837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47701d4d79e2464798024ea4a40e0377: Generated new TSK 0
I20260810 23:05:45.071707 16837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:45.085968 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 16706
W20260810 23:05:45.103252 16683 connection.cc:535] client connection to 127.0.0.1:43489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43489): Connection reset by peer (error 104)
2026-08-10T23:05:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1603 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:45 dist-test-slave-6bbx krb5kdc[16853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:45 dist-test-slave-6bbx krb5kdc[16853](info): set up 2 sockets
Aug 10 23:05:45 dist-test-slave-6bbx krb5kdc[16853](info): commencing operation
krb5kdc: starting...
W20260810 23:05:46.265877 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:46 dist-test-slave-6bbx krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:46.400895 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:46.523396 16877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:46.523797 16877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:46.523864 16877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:46.528178 16877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:46.528254 16877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:46.528272 16877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:46.528288 16877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:46.528302 16877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:46.534129 16877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:46.535444 16877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:46.536700 16877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:05:46.547439 16877 server_base.cc:1034] running on GCE node
W20260810 23:05:46.547456 16887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:46.547483 16894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:46.547690 16892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:46.548506 16877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:46.549855 16877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:46.551077 16877 hybrid_clock.cc:648] HybridClock initialized: now 1786403146551047 us; error 55 us; skew 500 ppm
Aug 10 23:05:46 dist-test-slave-6bbx krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:46.554533 16877 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:46.555655 16877 webserver.cc:460] Webserver started at http://127.0.0.1:34199/ using document root <none> and password file <none>
I20260810 23:05:46.556331 16877 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:46.556402 16877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:46.556607 16877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:46.559108 16877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "aabde1159e784ddfa0ebeb9d295c9790"
format_stamp: "Formatted at 2026-08-10 23:05:46 on dist-test-slave-6bbx"
server_key: "44da6c31a33117399c805bb238529f25"
server_key_iv: "a5009dd7565c12eda43e875ad1509185"
server_key_version: "encryptionkey@0"
I20260810 23:05:46.559798 16877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "aabde1159e784ddfa0ebeb9d295c9790"
format_stamp: "Formatted at 2026-08-10 23:05:46 on dist-test-slave-6bbx"
server_key: "44da6c31a33117399c805bb238529f25"
server_key_iv: "a5009dd7565c12eda43e875ad1509185"
server_key_version: "encryptionkey@0"
I20260810 23:05:46.564644 16877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260810 23:05:46.567874 16905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:46.569669 16877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:05:46.569836 16877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "aabde1159e784ddfa0ebeb9d295c9790"
format_stamp: "Formatted at 2026-08-10 23:05:46 on dist-test-slave-6bbx"
server_key: "44da6c31a33117399c805bb238529f25"
server_key_iv: "a5009dd7565c12eda43e875ad1509185"
server_key_version: "encryptionkey@0"
I20260810 23:05:46.569970 16877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:46.575206 16877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:46.578977 16877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:46.579160 16877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:46.587004 16877 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34411
I20260810 23:05:46.587080 16976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34411 every 8 connection(s)
I20260810 23:05:46.588223 16877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:46.589241 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 16877
I20260810 23:05:46.589427 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:46.589712 15584 external_mini_cluster.cc:1428] Setting key 6ef0461b891b3d13b6aa71981278b50f
I20260810 23:05:46.592020 16977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:46.595820 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:46.591483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:58253 (local address 127.0.0.1:34411)
0810 23:05:46.591997 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:46.592021 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:46.592076 (+ 55us) server_negotiation.cc:407] Connection header received
0810 23:05:46.592914 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:46.592936 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:46.593270 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:46.593624 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:46.594478 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:46.594483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:46.594496 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:46.594507 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:46.594566 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:46.594587 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:46.594729 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:46.594803 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 23:05:46.594990 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":169,"threads_started":1}
I20260810 23:05:46.598615 16977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790: Bootstrap starting.
I20260810 23:05:46.598824 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:46.601207 16977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:46.602212 16977 log.cc:824] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:46.602716 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:46.600991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:60467 (local address 127.0.0.1:34411)
0810 23:05:46.601129 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:46.601133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:46.601326 (+ 193us) server_negotiation.cc:407] Connection header received
0810 23:05:46.601475 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:46.601478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:46.601512 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:46.601579 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:46.602299 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:46.602302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:46.602304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:46.602314 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:46.602375 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:46.602390 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:46.602462 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:46.602556 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 23:05:46.602592 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:05:46.604599 16977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790: No bootstrap required, opened a new log
I20260810 23:05:46.608721 16977 raft_consensus.cc:348] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aabde1159e784ddfa0ebeb9d295c9790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260810 23:05:46.609028 16977 raft_consensus.cc:374] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:46.609066 16977 raft_consensus.cc:729] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aabde1159e784ddfa0ebeb9d295c9790, State: Initialized, Role: FOLLOWER
I20260810 23:05:46.609575 16977 consensus_queue.cc:260] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aabde1159e784ddfa0ebeb9d295c9790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260810 23:05:46.609714 16977 raft_consensus.cc:388] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:46.609771 16977 raft_consensus.cc:482] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:46.609879 16977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:46.610945 16977 raft_consensus.cc:504] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aabde1159e784ddfa0ebeb9d295c9790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260810 23:05:46.611397 16977 leader_election.cc:302] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aabde1159e784ddfa0ebeb9d295c9790; no voters:
I20260810 23:05:46.611878 16977 leader_election.cc:288] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:46.611997 16994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:46.612275 16994 raft_consensus.cc:686] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [term 1 LEADER]: Becoming Leader. State: Replica: aabde1159e784ddfa0ebeb9d295c9790, State: Running, Role: LEADER
I20260810 23:05:46.612671 16994 consensus_queue.cc:237] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aabde1159e784ddfa0ebeb9d295c9790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } }
I20260810 23:05:46.613241 16977 sys_catalog.cc:564] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:46.614481 16997 sys_catalog.cc:455] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aabde1159e784ddfa0ebeb9d295c9790. Latest consensus state: current_term: 1 leader_uuid: "aabde1159e784ddfa0ebeb9d295c9790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aabde1159e784ddfa0ebeb9d295c9790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } } }
I20260810 23:05:46.614481 16995 sys_catalog.cc:455] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aabde1159e784ddfa0ebeb9d295c9790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aabde1159e784ddfa0ebeb9d295c9790" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34411 } } }
I20260810 23:05:46.614638 16997 sys_catalog.cc:458] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:46.614638 16995 sys_catalog.cc:458] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:46.615064 17006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:46.618643 17006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:46.625387 17006 catalog_manager.cc:1347] Generated new cluster ID: 65b84810c1924adba89219527dcb6848
I20260810 23:05:46.625475 17006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:46.645239 17006 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:46.645601 17006 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:46.656122 17006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aabde1159e784ddfa0ebeb9d295c9790: Generated new TSK 0
I20260810 23:05:46.656915 17006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:46.660810 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 16877
2026-08-10T23:05:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1579 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:46 dist-test-slave-6bbx krb5kdc[17023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:46 dist-test-slave-6bbx krb5kdc[17023](info): set up 2 sockets
Aug 10 23:05:46 dist-test-slave-6bbx krb5kdc[17023](info): commencing operation
krb5kdc: starting...
W20260810 23:05:47.846184 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:47 dist-test-slave-6bbx krb5kdc[17023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:47.983350 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:48.105373 17050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:48.105717 17050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:48.105782 17050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:48.109965 17050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:48.110050 17050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:48.110077 17050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:48.110097 17050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:48.110117 17050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:48.115990 17050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44837
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:48.117329 17050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:48.118484 17050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:48.127807 17059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:48.127821 17055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:48.127851 17050 server_base.cc:1034] running on GCE node
W20260810 23:05:48.127807 17056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:48.128830 17050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:48.130128 17050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:48.131349 17050 hybrid_clock.cc:648] HybridClock initialized: now 1786403148131329 us; error 34 us; skew 500 ppm
Aug 10 23:05:48 dist-test-slave-6bbx krb5kdc[17023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:48.139266 17050 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:48.140425 17050 webserver.cc:460] Webserver started at http://127.0.0.1:45893/ using document root <none> and password file <none>
I20260810 23:05:48.141074 17050 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:48.141147 17050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:48.141369 17050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:48.143627 17050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "b896739ff6924559801c6a0202eb01d6"
format_stamp: "Formatted at 2026-08-10 23:05:48 on dist-test-slave-6bbx"
server_key: "48e21b37141c0588cd5a44fd2650c68a"
server_key_iv: "a4f29cb85dde6b16d5cd0c8d66eb204c"
server_key_version: "encryptionkey@0"
I20260810 23:05:48.144280 17050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "b896739ff6924559801c6a0202eb01d6"
format_stamp: "Formatted at 2026-08-10 23:05:48 on dist-test-slave-6bbx"
server_key: "48e21b37141c0588cd5a44fd2650c68a"
server_key_iv: "a4f29cb85dde6b16d5cd0c8d66eb204c"
server_key_version: "encryptionkey@0"
I20260810 23:05:48.148485 17050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:48.151455 17069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:48.152851 17050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:48.152987 17050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "b896739ff6924559801c6a0202eb01d6"
format_stamp: "Formatted at 2026-08-10 23:05:48 on dist-test-slave-6bbx"
server_key: "48e21b37141c0588cd5a44fd2650c68a"
server_key_iv: "a4f29cb85dde6b16d5cd0c8d66eb204c"
server_key_version: "encryptionkey@0"
I20260810 23:05:48.153105 17050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:48.168413 17050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:48.171777 17050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:48.171970 17050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:48.179617 17050 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44837
I20260810 23:05:48.179651 17158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44837 every 8 connection(s)
I20260810 23:05:48.180711 17050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:48.181684 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 17050
I20260810 23:05:48.181842 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:48.182180 15584 external_mini_cluster.cc:1428] Setting key 62c8311d3e362fa2e7706ed70c7aeca0
I20260810 23:05:48.184410 17159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:48.192025 17159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6: Bootstrap starting.
Aug 10 23:05:48 dist-test-slave-6bbx krb5kdc[17023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:05:48.194649 17159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:48.195443 17159 log.cc:824] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:48.197772 17159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6: No bootstrap required, opened a new log
I20260810 23:05:48.199810 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:48.183626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:33597 (local address 127.0.0.1:44837)
0810 23:05:48.186029 (+ 2403us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:48.186048 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:48.186094 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:05:48.187056 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:48.187082 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:48.187599 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:48.188055 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:48.188847 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:48.190378 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:48.191306 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:48.192036 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:48.192217 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:48.194905 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:48.194911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:48.194926 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:48.194958 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:48.196785 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:48.197305 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:48.197311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:48.197318 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:48.197441 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:48.197777 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:48.197781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:48.197783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:48.198219 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:48.198423 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:48.198756 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 23:05:48.198978 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2243,"thread_start_us":2014,"threads_started":1}
I20260810 23:05:48.201848 17159 raft_consensus.cc:348] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b896739ff6924559801c6a0202eb01d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } }
I20260810 23:05:48.202163 17159 raft_consensus.cc:374] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:48.202229 17159 raft_consensus.cc:729] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b896739ff6924559801c6a0202eb01d6, State: Initialized, Role: FOLLOWER
I20260810 23:05:48.202736 17159 consensus_queue.cc:260] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b896739ff6924559801c6a0202eb01d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } }
I20260810 23:05:48.202916 17159 raft_consensus.cc:388] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:48.202991 17159 raft_consensus.cc:482] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:48.203104 17159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:48.203183 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:48.204202 17159 raft_consensus.cc:504] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b896739ff6924559801c6a0202eb01d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } }
I20260810 23:05:48.204592 17159 leader_election.cc:302] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b896739ff6924559801c6a0202eb01d6; no voters:
I20260810 23:05:48.204955 17159 leader_election.cc:288] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:48.205436 17178 sys_catalog.cc:455] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b896739ff6924559801c6a0202eb01d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } } }
I20260810 23:05:48.205591 17178 sys_catalog.cc:458] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:05:48.205596 17174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:48.205775 17174 raft_consensus.cc:686] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [term 1 LEADER]: Becoming Leader. State: Replica: b896739ff6924559801c6a0202eb01d6, State: Running, Role: LEADER
I20260810 23:05:48.206171 17174 consensus_queue.cc:237] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b896739ff6924559801c6a0202eb01d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } }
I20260810 23:05:48.206425 17159 sys_catalog.cc:564] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:48.208050 17178 sys_catalog.cc:455] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b896739ff6924559801c6a0202eb01d6. Latest consensus state: current_term: 1 leader_uuid: "b896739ff6924559801c6a0202eb01d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b896739ff6924559801c6a0202eb01d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44837 } } }
I20260810 23:05:48.208250 17178 sys_catalog.cc:458] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6 [sys.catalog]: This master's current role is: LEADER
W20260810 23:05:48.212008 17194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:05:48.212150 17194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:48.212230 17195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:48.213146 17195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:48.215768 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:48.205367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:57641 (local address 127.0.0.1:44837)
0810 23:05:48.205480 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:48.205485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:48.205989 (+ 504us) server_negotiation.cc:407] Connection header received
0810 23:05:48.206133 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:48.206136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:48.206192 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:48.206281 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:48.207125 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:48.207930 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:48.209206 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:48.209530 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:48.209642 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:48.211396 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:48.211400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:48.211403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:48.211433 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:48.213495 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:48.214141 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:48.214145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:48.214147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:48.214255 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:48.214654 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:48.214657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:48.214659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:48.214933 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:48.215081 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:48.215404 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 23:05:48.215523 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:05:48.219981 17195 catalog_manager.cc:1347] Generated new cluster ID: cde58aeb209845179eb413c8e0261ecf
I20260810 23:05:48.220058 17195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:48.236559 17195 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:48.237596 17195 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:48.247184 17195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b896739ff6924559801c6a0202eb01d6: Generated new TSK 0
I20260810 23:05:48.248071 17195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:48.254089 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 17050
W20260810 23:05:48.277066 17020 connection.cc:535] client connection to 127.0.0.1:44837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44837): Connection reset by peer (error 104)
2026-08-10T23:05:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1594 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:48 dist-test-slave-6bbx krb5kdc[17210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:48 dist-test-slave-6bbx krb5kdc[17210](info): set up 2 sockets
Aug 10 23:05:48 dist-test-slave-6bbx krb5kdc[17210](info): commencing operation
krb5kdc: starting...
W20260810 23:05:49.435952 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:49 dist-test-slave-6bbx krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:49.573043 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:49.691598 17238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:49.691946 17238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:49.691990 17238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:49.696161 17238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:49.696238 17238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:49.696257 17238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:49.696272 17238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:49.696286 17238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:49.701805 17238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:49.702980 17238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:49.704141 17238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:49.713063 17258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:49.713055 17245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:49.713111 17238 server_base.cc:1034] running on GCE node
W20260810 23:05:49.713057 17248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:49.713927 17238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:49.715044 17238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:49.716228 17238 hybrid_clock.cc:648] HybridClock initialized: now 1786403149716201 us; error 38 us; skew 500 ppm
Aug 10 23:05:49 dist-test-slave-6bbx krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:49.719172 17238 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:49.720182 17238 webserver.cc:460] Webserver started at http://127.0.0.1:37969/ using document root <none> and password file <none>
I20260810 23:05:49.720749 17238 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:49.720809 17238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:49.721014 17238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:49.723109 17238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "6ed4d4dffe1f483eb8d51f2856ab3e3a"
format_stamp: "Formatted at 2026-08-10 23:05:49 on dist-test-slave-6bbx"
server_key: "40d6d58d39f6ee3e6fd31ecd23d91602"
server_key_iv: "95d464140d7a4331c6db78ab5a9807af"
server_key_version: "encryptionkey@0"
I20260810 23:05:49.723655 17238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "6ed4d4dffe1f483eb8d51f2856ab3e3a"
format_stamp: "Formatted at 2026-08-10 23:05:49 on dist-test-slave-6bbx"
server_key: "40d6d58d39f6ee3e6fd31ecd23d91602"
server_key_iv: "95d464140d7a4331c6db78ab5a9807af"
server_key_version: "encryptionkey@0"
I20260810 23:05:49.727483 17238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 23:05:49.730024 17268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:49.731331 17238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:05:49.731457 17238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "6ed4d4dffe1f483eb8d51f2856ab3e3a"
format_stamp: "Formatted at 2026-08-10 23:05:49 on dist-test-slave-6bbx"
server_key: "40d6d58d39f6ee3e6fd31ecd23d91602"
server_key_iv: "95d464140d7a4331c6db78ab5a9807af"
server_key_version: "encryptionkey@0"
I20260810 23:05:49.731568 17238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:49.750764 17238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:49.754535 17238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:49.754737 17238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:49.762218 17238 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35115
I20260810 23:05:49.762218 17364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35115 every 8 connection(s)
I20260810 23:05:49.763273 17238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:49.766213 17365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:49.771113 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 17238
I20260810 23:05:49.771258 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:49.771538 15584 external_mini_cluster.cc:1428] Setting key 6afcffa713dcc41445f934e709f33c28
I20260810 23:05:49.772399 17365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a: Bootstrap starting.
I20260810 23:05:49.774937 17365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:49.775851 17365 log.cc:824] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:49.778178 17365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a: No bootstrap required, opened a new log
I20260810 23:05:49.781534 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:49.772995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:59347 (local address 127.0.0.1:35115)
0810 23:05:49.773441 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:49.773464 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:49.773502 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:05:49.774189 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:49.774207 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:49.774581 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:49.774881 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:49.775302 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:49.776410 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:49.777382 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:49.778081 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:49.778272 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:49.779180 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:49.779187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:49.779207 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:49.779809 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:49.779883 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:49.779917 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:49.780059 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:49.780302 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 23:05:49.780455 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":125,"threads_started":1}
I20260810 23:05:49.782697 17365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed4d4dffe1f483eb8d51f2856ab3e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } }
I20260810 23:05:49.782935 17365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:49.782982 17365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed4d4dffe1f483eb8d51f2856ab3e3a, State: Initialized, Role: FOLLOWER
I20260810 23:05:49.783427 17365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed4d4dffe1f483eb8d51f2856ab3e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } }
I20260810 23:05:49.783588 17365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:49.783726 17365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:49.783844 17365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:49.784792 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:49.784874 17365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed4d4dffe1f483eb8d51f2856ab3e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } }
I20260810 23:05:49.785254 17365 leader_election.cc:302] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ed4d4dffe1f483eb8d51f2856ab3e3a; no voters:
I20260810 23:05:49.785660 17365 leader_election.cc:288] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:49.785732 17372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:49.786010 17372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [term 1 LEADER]: Becoming Leader. State: Replica: 6ed4d4dffe1f483eb8d51f2856ab3e3a, State: Running, Role: LEADER
I20260810 23:05:49.786391 17372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed4d4dffe1f483eb8d51f2856ab3e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } }
I20260810 23:05:49.787120 17365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:49.788123 17373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ed4d4dffe1f483eb8d51f2856ab3e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed4d4dffe1f483eb8d51f2856ab3e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } } }
I20260810 23:05:49.788127 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ed4d4dffe1f483eb8d51f2856ab3e3a. Latest consensus state: current_term: 1 leader_uuid: "6ed4d4dffe1f483eb8d51f2856ab3e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed4d4dffe1f483eb8d51f2856ab3e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35115 } } }
I20260810 23:05:49.788278 17373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:49.788278 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:49.788650 17394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:49.792178 17394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:49.793313 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:49.787150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:46759 (local address 127.0.0.1:35115)
0810 23:05:49.787263 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:49.787267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:49.788688 (+ 1421us) server_negotiation.cc:407] Connection header received
0810 23:05:49.788886 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:49.788889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:49.788937 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:49.788984 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:49.789420 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:49.790153 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:49.791351 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:49.791710 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:49.791815 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:49.792784 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:49.792788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:49.792792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:49.792806 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:49.792837 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:49.792851 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:49.792918 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:49.793137 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 23:05:49.793174 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 23:05:49.797982 17394 catalog_manager.cc:1347] Generated new cluster ID: bdd586eaa4774044b60d298d2942e84a
I20260810 23:05:49.798051 17394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:49.818217 17394 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:49.819332 17394 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:49.833036 17394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ed4d4dffe1f483eb8d51f2856ab3e3a: Generated new TSK 0
I20260810 23:05:49.833849 17394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:49.845402 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 17238
W20260810 23:05:49.860651 17207 connection.cc:535] client connection to 127.0.0.1:35115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35115): Connection reset by peer (error 104)
2026-08-10T23:05:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1583 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:49 dist-test-slave-6bbx krb5kdc[17408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:49 dist-test-slave-6bbx krb5kdc[17408](info): set up 2 sockets
Aug 10 23:05:49 dist-test-slave-6bbx krb5kdc[17408](info): commencing operation
krb5kdc: starting...
W20260810 23:05:51.040823 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:51 dist-test-slave-6bbx krb5kdc[17408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:05:51.174466 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:51.289083 17430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:51.289358 17430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:51.289400 17430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:51.293442 17430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:51.293506 17430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:51.293524 17430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:51.293540 17430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:51.293552 17430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:51.298848 17430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:51.299945 17430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:51.301060 17430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:51.310073 17440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:51.310088 17438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:51.310302 17442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:51.310398 17430 server_base.cc:1034] running on GCE node
I20260810 23:05:51.311058 17430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:51.312331 17430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:51.313527 17430 hybrid_clock.cc:648] HybridClock initialized: now 1786403151313500 us; error 40 us; skew 500 ppm
Aug 10 23:05:51 dist-test-slave-6bbx krb5kdc[17408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:51.316545 17430 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:05:51.317550 17430 webserver.cc:460] Webserver started at http://127.0.0.1:43443/ using document root <none> and password file <none>
I20260810 23:05:51.318140 17430 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:51.318212 17430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:51.318425 17430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:51.320766 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "6069b7af54834226a8ad856e4159efa9"
format_stamp: "Formatted at 2026-08-10 23:05:51 on dist-test-slave-6bbx"
server_key: "c86df8d62403f2e53e3205f777feb254"
server_key_iv: "54e2d284b7bdc7148d859c8f9d7a26c6"
server_key_version: "encryptionkey@0"
I20260810 23:05:51.321373 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "6069b7af54834226a8ad856e4159efa9"
format_stamp: "Formatted at 2026-08-10 23:05:51 on dist-test-slave-6bbx"
server_key: "c86df8d62403f2e53e3205f777feb254"
server_key_iv: "54e2d284b7bdc7148d859c8f9d7a26c6"
server_key_version: "encryptionkey@0"
I20260810 23:05:51.325470 17430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:05:51.328294 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:51.329627 17430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:51.329767 17430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "6069b7af54834226a8ad856e4159efa9"
format_stamp: "Formatted at 2026-08-10 23:05:51 on dist-test-slave-6bbx"
server_key: "c86df8d62403f2e53e3205f777feb254"
server_key_iv: "54e2d284b7bdc7148d859c8f9d7a26c6"
server_key_version: "encryptionkey@0"
I20260810 23:05:51.329877 17430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:51.334894 17430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:51.338135 17430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:51.338315 17430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:51.345239 17430 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41179
I20260810 23:05:51.345285 17541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41179 every 8 connection(s)
I20260810 23:05:51.346252 17430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:51.349892 17542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:51.351979 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 17430
I20260810 23:05:51.352133 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:51.352468 15584 external_mini_cluster.cc:1428] Setting key e247d2fc0e29d8cf14182fdd5dd4987e
I20260810 23:05:51.357136 17542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9: Bootstrap starting.
I20260810 23:05:51.357951 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.353945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:49053 (local address 127.0.0.1:41179)
0810 23:05:51.354400 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:51.354416 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:51.354456 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:05:51.355106 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:51.355124 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:51.355436 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:51.355768 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:51.356605 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:51.356611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:51.356629 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:51.356641 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:51.356703 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:51.356732 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:51.356853 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:51.356965 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 23:05:51.357173 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":136,"threads_started":1}
I20260810 23:05:51.359632 17542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:51.360504 17542 log.cc:824] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:51.361140 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:51.362679 17542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9: No bootstrap required, opened a new log
I20260810 23:05:51.364858 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.363224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.28:35855 (local address 127.0.0.1:41179)
0810 23:05:51.363333 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:51.363337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:51.363509 (+ 172us) server_negotiation.cc:407] Connection header received
0810 23:05:51.363623 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:51.363626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:51.363682 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:51.363755 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:51.364462 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:51.364465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:51.364468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:51.364479 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:51.364507 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:51.364517 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:51.364589 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:51.364699 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 23:05:51.364742 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 23:05:51.366283 17542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6069b7af54834226a8ad856e4159efa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41179 } }
I20260810 23:05:51.366513 17542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:51.366554 17542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6069b7af54834226a8ad856e4159efa9, State: Initialized, Role: FOLLOWER
I20260810 23:05:51.366942 17542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6069b7af54834226a8ad856e4159efa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41179 } }
I20260810 23:05:51.367057 17542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:51.367103 17542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:51.367178 17542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:51.368139 17542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6069b7af54834226a8ad856e4159efa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41179 } }
I20260810 23:05:51.368445 17542 leader_election.cc:302] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6069b7af54834226a8ad856e4159efa9; no voters:
I20260810 23:05:51.368749 17542 leader_election.cc:288] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:51.368839 17557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:51.369024 17557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [term 1 LEADER]: Becoming Leader. State: Replica: 6069b7af54834226a8ad856e4159efa9, State: Running, Role: LEADER
I20260810 23:05:51.369362 17557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6069b7af54834226a8ad856e4159efa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41179 } }
I20260810 23:05:51.369931 17542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:51.371129 17560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6069b7af54834226a8ad856e4159efa9. Latest consensus state: current_term: 1 leader_uuid: "6069b7af54834226a8ad856e4159efa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6069b7af54834226a8ad856e4159efa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41179 } } }
I20260810 23:05:51.371178 17559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6069b7af54834226a8ad856e4159efa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6069b7af54834226a8ad856e4159efa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41179 } } }
I20260810 23:05:51.371309 17559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:51.371579 17560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:51.371711 17567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:51.374778 17567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:51.380232 17567 catalog_manager.cc:1347] Generated new cluster ID: 61c31a8edf2845c584c8d72f3cface13
I20260810 23:05:51.380309 17567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:51.394892 17567 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:51.395294 17567 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:51.405912 17567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6069b7af54834226a8ad856e4159efa9: Generated new TSK 0
I20260810 23:05:51.406893 17567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:51.420837 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 17430
2026-08-10T23:05:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1572 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:51 dist-test-slave-6bbx krb5kdc[17585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:51 dist-test-slave-6bbx krb5kdc[17585](info): set up 2 sockets
Aug 10 23:05:51 dist-test-slave-6bbx krb5kdc[17585](info): commencing operation
krb5kdc: starting...
W20260810 23:05:52.584545 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:52 dist-test-slave-6bbx krb5kdc[17585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:05:52.714181 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43291
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:43291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:52.832118 17607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:52.832443 17607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:52.832489 17607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:52.836622 17607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:52.836692 17607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:52.836711 17607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:52.836726 17607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:52.836740 17607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:52.842376 17607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:43291
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:52.843487 17607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:52.844571 17607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:05:52.853827 17607 server_base.cc:1034] running on GCE node
W20260810 23:05:52.853713 17615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:52.853724 17614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:52.853757 17618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:52.854627 17607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:52.855885 17607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:52.857123 17607 hybrid_clock.cc:648] HybridClock initialized: now 1786403152857089 us; error 41 us; skew 500 ppm
Aug 10 23:05:52 dist-test-slave-6bbx krb5kdc[17585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403152, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:52.860360 17607 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:05:52.861413 17607 webserver.cc:460] Webserver started at http://127.15.56.62:43685/ using document root <none> and password file <none>
I20260810 23:05:52.862002 17607 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:52.862075 17607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:52.862336 17607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:52.864639 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "de0859b7675b4e7a8247144df01e236e"
format_stamp: "Formatted at 2026-08-10 23:05:52 on dist-test-slave-6bbx"
server_key: "95bb02287bc8f751ab91078bb131c383"
server_key_iv: "6c4fa9497cc68929a08eab31139c9786"
server_key_version: "encryptionkey@0"
I20260810 23:05:52.865259 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "de0859b7675b4e7a8247144df01e236e"
format_stamp: "Formatted at 2026-08-10 23:05:52 on dist-test-slave-6bbx"
server_key: "95bb02287bc8f751ab91078bb131c383"
server_key_iv: "6c4fa9497cc68929a08eab31139c9786"
server_key_version: "encryptionkey@0"
I20260810 23:05:52.869504 17607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:05:52.872222 17629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:52.873658 17607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:52.873807 17607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "de0859b7675b4e7a8247144df01e236e"
format_stamp: "Formatted at 2026-08-10 23:05:52 on dist-test-slave-6bbx"
server_key: "95bb02287bc8f751ab91078bb131c383"
server_key_iv: "6c4fa9497cc68929a08eab31139c9786"
server_key_version: "encryptionkey@0"
I20260810 23:05:52.873926 17607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:52.891779 17607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:52.894804 17607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:52.895010 17607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:52.902463 17607 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:43291
I20260810 23:05:52.902490 17725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:43291 every 8 connection(s)
I20260810 23:05:52.903483 17607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:52.906591 17726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:52.911988 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 17607
I20260810 23:05:52.912125 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:52.912410 15584 external_mini_cluster.cc:1428] Setting key bf91280251e2dd7b81bb2da19b1be9a9
I20260810 23:05:52.913097 17726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e: Bootstrap starting.
I20260810 23:05:52.915940 17726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:52.916855 17726 log.cc:824] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:52.919083 17726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e: No bootstrap required, opened a new log
Aug 10 23:05:52 dist-test-slave-6bbx krb5kdc[17585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:05:52.922901 17726 raft_consensus.cc:348] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0859b7675b4e7a8247144df01e236e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43291 } }
I20260810 23:05:52.923141 17726 raft_consensus.cc:374] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:52.923198 17726 raft_consensus.cc:729] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de0859b7675b4e7a8247144df01e236e, State: Initialized, Role: FOLLOWER
I20260810 23:05:52.923686 17726 consensus_queue.cc:260] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0859b7675b4e7a8247144df01e236e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43291 } }
I20260810 23:05:52.923844 17726 raft_consensus.cc:388] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:52.923911 17726 raft_consensus.cc:482] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:52.924009 17726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:52.924889 17726 raft_consensus.cc:504] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0859b7675b4e7a8247144df01e236e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43291 } }
I20260810 23:05:52.925215 17726 leader_election.cc:302] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de0859b7675b4e7a8247144df01e236e; no voters:
I20260810 23:05:52.925537 17726 leader_election.cc:288] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:52.925621 17731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:52.925809 17731 raft_consensus.cc:686] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [term 1 LEADER]: Becoming Leader. State: Replica: de0859b7675b4e7a8247144df01e236e, State: Running, Role: LEADER
I20260810 23:05:52.926177 17731 consensus_queue.cc:237] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0859b7675b4e7a8247144df01e236e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43291 } }
I20260810 23:05:52.926872 17726 sys_catalog.cc:564] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:52.927896 17735 sys_catalog.cc:455] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de0859b7675b4e7a8247144df01e236e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0859b7675b4e7a8247144df01e236e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43291 } } }
I20260810 23:05:52.928020 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:52.913941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34671 (local address 127.15.56.62:43291)
0810 23:05:52.914414 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:52.914433 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:52.914477 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:05:52.915345 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:52.915370 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:52.915877 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:52.916296 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:52.916970 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:52.918285 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:52.919212 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:52.919780 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:52.919938 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:52.922607 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:52.922612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:52.922626 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:52.922653 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:52.924429 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:52.924908 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:52.924914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:52.924921 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:52.925048 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:52.925374 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:52.925378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:52.925380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:52.925878 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:52.926651 (+ 773us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:52.926806 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:05:52.927012 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":135,"threads_started":1}
I20260810 23:05:52.928025 17735 sys_catalog.cc:458] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:52.928099 17732 sys_catalog.cc:455] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [sys.catalog]: SysCatalogTable state changed. Reason: New leader de0859b7675b4e7a8247144df01e236e. Latest consensus state: current_term: 1 leader_uuid: "de0859b7675b4e7a8247144df01e236e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de0859b7675b4e7a8247144df01e236e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43291 } } }
I20260810 23:05:52.928160 17732 sys_catalog.cc:458] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:52.928467 17744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:52.931130 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:52.931617 17744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:52.937017 17744 catalog_manager.cc:1347] Generated new cluster ID: dc0c39177137478292c48bce44e03024
I20260810 23:05:52.937129 17744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:52.943125 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:52.933392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59455 (local address 127.15.56.62:43291)
0810 23:05:52.933486 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:52.933490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:52.933989 (+ 499us) server_negotiation.cc:407] Connection header received
0810 23:05:52.934172 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:52.934175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:52.934222 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:52.934274 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:52.935185 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:52.935859 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:52.937045 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:52.937439 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:52.937514 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:52.939386 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:52.939390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:52.939392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:52.939418 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:52.940939 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:52.941523 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:52.941527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:52.941529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:52.941618 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:52.942026 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:52.942029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:52.942032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:52.942268 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:52.942391 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:52.942851 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 23:05:52.942955 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 23:05:52.964841 17744 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:52.965822 17744 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:52.970305 17744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de0859b7675b4e7a8247144df01e236e: Generated new TSK 0
I20260810 23:05:52.971215 17744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:52.977586 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 17607
W20260810 23:05:52.992162 17583 connection.cc:535] client connection to 127.15.56.62:43291 recv error: Network error: failed to read from TLS socket (remote: 127.15.56.62:43291): Connection reset by peer (error 104)
2026-08-10T23:05:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1558 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:53 dist-test-slave-6bbx krb5kdc[17768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:53 dist-test-slave-6bbx krb5kdc[17768](info): set up 2 sockets
Aug 10 23:05:53 dist-test-slave-6bbx krb5kdc[17768](info): commencing operation
krb5kdc: starting...
W20260810 23:05:54.141026 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:54 dist-test-slave-6bbx krb5kdc[17768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:05:54.274396 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43329
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:54.388365 17787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:54.388659 17787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:54.388706 17787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:54.392725 17787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:54.392786 17787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:54.392803 17787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:54.392817 17787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:54.392831 17787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:54.398156 17787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:43329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:54.399266 17787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:54.400394 17787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:54.409400 17795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:54.409415 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:54.409390 17797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:54.409657 17787 server_base.cc:1034] running on GCE node
I20260810 23:05:54.410334 17787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:54.411494 17787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:54.412660 17787 hybrid_clock.cc:648] HybridClock initialized: now 1786403154412649 us; error 41 us; skew 500 ppm
Aug 10 23:05:54 dist-test-slave-6bbx krb5kdc[17768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403154, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:54.419083 17787 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:05:54.420161 17787 webserver.cc:460] Webserver started at http://127.15.56.62:44693/ using document root <none> and password file <none>
I20260810 23:05:54.420754 17787 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:54.420838 17787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:54.421068 17787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:54.423336 17787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "de16a3a2640c44b1ac12363a7726945e"
format_stamp: "Formatted at 2026-08-10 23:05:54 on dist-test-slave-6bbx"
server_key: "c9629a50e941b3c70f3ca5f1170cbc2a"
server_key_iv: "8b894e4a395505672663f2a2f3fff15f"
server_key_version: "encryptionkey@0"
I20260810 23:05:54.423933 17787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "de16a3a2640c44b1ac12363a7726945e"
format_stamp: "Formatted at 2026-08-10 23:05:54 on dist-test-slave-6bbx"
server_key: "c9629a50e941b3c70f3ca5f1170cbc2a"
server_key_iv: "8b894e4a395505672663f2a2f3fff15f"
server_key_version: "encryptionkey@0"
I20260810 23:05:54.427922 17787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:05:54.430574 17811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:54.432013 17787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:54.432138 17787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "de16a3a2640c44b1ac12363a7726945e"
format_stamp: "Formatted at 2026-08-10 23:05:54 on dist-test-slave-6bbx"
server_key: "c9629a50e941b3c70f3ca5f1170cbc2a"
server_key_iv: "8b894e4a395505672663f2a2f3fff15f"
server_key_version: "encryptionkey@0"
I20260810 23:05:54.432248 17787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:54.448904 17787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:54.452222 17787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:54.452425 17787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:54.459452 17787 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:43329
I20260810 23:05:54.459524 17888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:43329 every 8 connection(s)
I20260810 23:05:54.460649 17787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:54.461755 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 17787
I20260810 23:05:54.461901 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:54.462190 15584 external_mini_cluster.cc:1428] Setting key e348b07ac36b99ed25168fdb3d269600
I20260810 23:05:54.463929 17891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:54.471051 17891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e: Bootstrap starting.
I20260810 23:05:54.472357 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:54.463622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58259 (local address 127.15.56.62:43329)
0810 23:05:54.464113 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:54.464133 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:54.464176 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:05:54.465028 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:54.465053 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:54.465490 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:54.465874 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:54.466277 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:54.467806 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:54.468711 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:54.469399 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:54.469562 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:54.470358 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:54.470365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:54.470381 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:54.470886 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:54.470955 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:54.470986 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:54.471122 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:54.471264 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 23:05:54.471399 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":145,"threads_started":1}
I20260810 23:05:54.473636 17891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:54.474407 17891 log.cc:824] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:54.475234 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:54.476440 17891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e: No bootstrap required, opened a new log
I20260810 23:05:54.479542 17891 raft_consensus.cc:348] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16a3a2640c44b1ac12363a7726945e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43329 } }
I20260810 23:05:54.479815 17891 raft_consensus.cc:374] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:54.479878 17891 raft_consensus.cc:729] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de16a3a2640c44b1ac12363a7726945e, State: Initialized, Role: FOLLOWER
I20260810 23:05:54.480324 17891 consensus_queue.cc:260] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16a3a2640c44b1ac12363a7726945e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43329 } }
I20260810 23:05:54.480458 17891 raft_consensus.cc:388] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:54.480523 17891 raft_consensus.cc:482] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:54.480621 17891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:54.481417 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:54.477143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57795 (local address 127.15.56.62:43329)
0810 23:05:54.477265 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:54.477269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:54.477434 (+ 165us) server_negotiation.cc:407] Connection header received
0810 23:05:54.477540 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:54.477543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:54.477582 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:54.477647 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:54.477905 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:54.478916 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:54.479759 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:54.480073 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:54.480172 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:54.480965 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:54.480969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:54.480972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:54.480985 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:54.481011 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:54.481022 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:54.481087 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:54.481236 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 23:05:54.481273 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:05:54.481562 17891 raft_consensus.cc:504] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16a3a2640c44b1ac12363a7726945e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43329 } }
I20260810 23:05:54.481918 17891 leader_election.cc:302] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de16a3a2640c44b1ac12363a7726945e; no voters:
I20260810 23:05:54.482257 17891 leader_election.cc:288] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:54.482347 17909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:54.482523 17909 raft_consensus.cc:686] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [term 1 LEADER]: Becoming Leader. State: Replica: de16a3a2640c44b1ac12363a7726945e, State: Running, Role: LEADER
I20260810 23:05:54.482865 17909 consensus_queue.cc:237] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16a3a2640c44b1ac12363a7726945e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43329 } }
I20260810 23:05:54.483633 17891 sys_catalog.cc:564] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:54.484730 17910 sys_catalog.cc:455] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [sys.catalog]: SysCatalogTable state changed. Reason: New leader de16a3a2640c44b1ac12363a7726945e. Latest consensus state: current_term: 1 leader_uuid: "de16a3a2640c44b1ac12363a7726945e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16a3a2640c44b1ac12363a7726945e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43329 } } }
I20260810 23:05:54.485006 17913 sys_catalog.cc:455] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de16a3a2640c44b1ac12363a7726945e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de16a3a2640c44b1ac12363a7726945e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43329 } } }
I20260810 23:05:54.485122 17910 sys_catalog.cc:458] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:54.485237 17913 sys_catalog.cc:458] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:54.485677 17931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:54.488700 17931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:54.494511 17931 catalog_manager.cc:1347] Generated new cluster ID: e122d98c344542378883c6c77c597588
I20260810 23:05:54.494592 17931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:54.507892 17931 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:54.508889 17931 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:54.519487 17931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de16a3a2640c44b1ac12363a7726945e: Generated new TSK 0
I20260810 23:05:54.520337 17931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:54.527854 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 17787
W20260810 23:05:54.543607 17765 connection.cc:535] client connection to 127.15.56.62:43329 recv error: Network error: failed to read from TLS socket (remote: 127.15.56.62:43329): Connection reset by peer (error 104)
2026-08-10T23:05:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1551 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:54 dist-test-slave-6bbx krb5kdc[17941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:54 dist-test-slave-6bbx krb5kdc[17941](info): set up 2 sockets
Aug 10 23:05:54 dist-test-slave-6bbx krb5kdc[17941](info): commencing operation
krb5kdc: starting...
W20260810 23:05:55.681043 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:55 dist-test-slave-6bbx krb5kdc[17941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:05:55.817137 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:37717
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:37717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:55.936834 17958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:55.937160 17958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:55.937223 17958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:55.941391 17958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:55.941473 17958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:55.941501 17958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:55.941522 17958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:55.941542 17958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:55.947245 17958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:37717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:37717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:55.948520 17958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:55.949681 17958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:55.959209 17965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:55.959332 17970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:55.959406 17958 server_base.cc:1034] running on GCE node
W20260810 23:05:55.959210 17966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:55.960256 17958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:55.961483 17958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:55.962661 17958 hybrid_clock.cc:648] HybridClock initialized: now 1786403155962640 us; error 31 us; skew 500 ppm
Aug 10 23:05:55 dist-test-slave-6bbx krb5kdc[17941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403155, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:55.965778 17958 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:05:55.966845 17958 webserver.cc:460] Webserver started at http://127.15.56.62:32901/ using document root <none> and password file <none>
I20260810 23:05:55.967455 17958 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:55.967532 17958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:55.967815 17958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:55.970152 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "1c3c4acf73a44ec4bb847252fe4b49b0"
format_stamp: "Formatted at 2026-08-10 23:05:55 on dist-test-slave-6bbx"
server_key: "76a7bcd29b2e0bbc3d9a3b021f37784a"
server_key_iv: "30690f14eb44db51f77c013353c906d1"
server_key_version: "encryptionkey@0"
I20260810 23:05:55.970821 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "1c3c4acf73a44ec4bb847252fe4b49b0"
format_stamp: "Formatted at 2026-08-10 23:05:55 on dist-test-slave-6bbx"
server_key: "76a7bcd29b2e0bbc3d9a3b021f37784a"
server_key_iv: "30690f14eb44db51f77c013353c906d1"
server_key_version: "encryptionkey@0"
I20260810 23:05:55.975188 17958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:05:55.978082 17983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:55.979516 17958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:05:55.979735 17958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "1c3c4acf73a44ec4bb847252fe4b49b0"
format_stamp: "Formatted at 2026-08-10 23:05:55 on dist-test-slave-6bbx"
server_key: "76a7bcd29b2e0bbc3d9a3b021f37784a"
server_key_iv: "30690f14eb44db51f77c013353c906d1"
server_key_version: "encryptionkey@0"
I20260810 23:05:55.979866 17958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:55.984838 17958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:55.988423 17958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:55.988620 17958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:55.995798 17958 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:37717
I20260810 23:05:55.995839 18062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:37717 every 8 connection(s)
I20260810 23:05:55.996796 17958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:55.999857 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:56.005442 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 17958
I20260810 23:05:56.005627 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:56.005944 15584 external_mini_cluster.cc:1428] Setting key 5c8d96f8b104219617b01128351d5260
I20260810 23:05:56.006860 18063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0: Bootstrap starting.
I20260810 23:05:56.009721 18063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:56.010666 18063 log.cc:824] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:56.012143 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:56.007693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43761 (local address 127.15.56.62:37717)
0810 23:05:56.008315 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:56.008334 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:56.008380 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:05:56.009132 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:56.009165 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:56.009498 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:56.009869 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:56.010721 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:56.010726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:56.010741 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:56.010752 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:56.010818 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:56.010847 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:56.010953 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:56.011099 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 23:05:56.011288 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":166,"threads_started":1}
I20260810 23:05:56.012957 18063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0: No bootstrap required, opened a new log
I20260810 23:05:56.015224 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:56.017436 18063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c4acf73a44ec4bb847252fe4b49b0" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37717 } }
I20260810 23:05:56.017724 18063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:56.017802 18063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3c4acf73a44ec4bb847252fe4b49b0, State: Initialized, Role: FOLLOWER
I20260810 23:05:56.018368 18063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c4acf73a44ec4bb847252fe4b49b0" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37717 } }
I20260810 23:05:56.018579 18063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:56.018666 18063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:56.018784 18063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:56.019613 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:56.017820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47045 (local address 127.15.56.62:37717)
0810 23:05:56.017944 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:56.017948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:56.018168 (+ 220us) server_negotiation.cc:407] Connection header received
0810 23:05:56.018303 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:56.018305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:56.018342 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:56.018416 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:56.019227 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:56.019230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:56.019232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:56.019244 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:56.019276 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:56.019289 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:56.019364 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:56.019455 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 23:05:56.019491 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:05:56.020161 18063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c4acf73a44ec4bb847252fe4b49b0" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37717 } }
I20260810 23:05:56.020542 18063 leader_election.cc:302] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c3c4acf73a44ec4bb847252fe4b49b0; no voters:
I20260810 23:05:56.020885 18063 leader_election.cc:288] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:56.021030 18077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:56.021262 18077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [term 1 LEADER]: Becoming Leader. State: Replica: 1c3c4acf73a44ec4bb847252fe4b49b0, State: Running, Role: LEADER
I20260810 23:05:56.021610 18077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c4acf73a44ec4bb847252fe4b49b0" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37717 } }
I20260810 23:05:56.022213 18063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:56.023411 18078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c3c4acf73a44ec4bb847252fe4b49b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c4acf73a44ec4bb847252fe4b49b0" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37717 } } }
I20260810 23:05:56.023411 18079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c3c4acf73a44ec4bb847252fe4b49b0. Latest consensus state: current_term: 1 leader_uuid: "1c3c4acf73a44ec4bb847252fe4b49b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c3c4acf73a44ec4bb847252fe4b49b0" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 37717 } } }
I20260810 23:05:56.023608 18078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:56.023687 18079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0 [sys.catalog]: This master's current role is: LEADER
W20260810 23:05:56.027527 18100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:05:56.027706 18100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:56.027787 18093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:56.028533 18093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:56.034906 18093 catalog_manager.cc:1347] Generated new cluster ID: ba6f2ac4064a4dcb8a0ca7b8ffc9e41c
I20260810 23:05:56.034997 18093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:56.055528 18093 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:56.055938 18093 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:56.065757 18093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c3c4acf73a44ec4bb847252fe4b49b0: Generated new TSK 0
I20260810 23:05:56.066622 18093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:56.081022 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 17958
2026-08-10T23:05:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1553 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:56 dist-test-slave-6bbx krb5kdc[18110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:56 dist-test-slave-6bbx krb5kdc[18110](info): set up 2 sockets
Aug 10 23:05:56 dist-test-slave-6bbx krb5kdc[18110](info): commencing operation
krb5kdc: starting...
W20260810 23:05:57.251327 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:57 dist-test-slave-6bbx krb5kdc[18110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:05:57.387387 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:35511
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:35511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:57.509977 18133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:57.510337 18133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:57.510402 18133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:57.514635 18133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:57.514727 18133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:57.514755 18133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:57.514777 18133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:57.514799 18133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:57.520650 18133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:35511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:35511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:57.521919 18133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:57.523097 18133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:57.532959 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:57.532939 18141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:57.533002 18133 server_base.cc:1034] running on GCE node
W20260810 23:05:57.532939 18151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:57.533974 18133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:57.535297 18133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:57.536507 18133 hybrid_clock.cc:648] HybridClock initialized: now 1786403157536490 us; error 46 us; skew 500 ppm
Aug 10 23:05:57 dist-test-slave-6bbx krb5kdc[18110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403157, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:57.539986 18133 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:05:57.541137 18133 webserver.cc:460] Webserver started at http://127.15.56.62:33091/ using document root <none> and password file <none>
I20260810 23:05:57.541757 18133 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:57.541836 18133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:57.542057 18133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:57.544435 18133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "1b5667be4d544258a7f2fa41fe139663"
format_stamp: "Formatted at 2026-08-10 23:05:57 on dist-test-slave-6bbx"
server_key: "ede1967f92091e9036137a9e28ab2c87"
server_key_iv: "288a1b7677a61e6ff0aa6ae8e9436388"
server_key_version: "encryptionkey@0"
I20260810 23:05:57.545054 18133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "1b5667be4d544258a7f2fa41fe139663"
format_stamp: "Formatted at 2026-08-10 23:05:57 on dist-test-slave-6bbx"
server_key: "ede1967f92091e9036137a9e28ab2c87"
server_key_iv: "288a1b7677a61e6ff0aa6ae8e9436388"
server_key_version: "encryptionkey@0"
I20260810 23:05:57.549111 18133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:05:57.551952 18159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:57.553396 18133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:57.553542 18133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "1b5667be4d544258a7f2fa41fe139663"
format_stamp: "Formatted at 2026-08-10 23:05:57 on dist-test-slave-6bbx"
server_key: "ede1967f92091e9036137a9e28ab2c87"
server_key_iv: "288a1b7677a61e6ff0aa6ae8e9436388"
server_key_version: "encryptionkey@0"
I20260810 23:05:57.553661 18133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:57.570894 18133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:57.574381 18133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:57.574604 18133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:57.582423 18133 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:35511
I20260810 23:05:57.582456 18240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:35511 every 8 connection(s)
I20260810 23:05:57.583468 18133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:57.586109 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 18133
I20260810 23:05:57.586244 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:57.586537 15584 external_mini_cluster.cc:1428] Setting key c7cbbc55b82334ba1c3950b4028106ad
I20260810 23:05:57.586561 18241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:57.593349 18241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663: Bootstrap starting.
Aug 10 23:05:57 dist-test-slave-6bbx krb5kdc[18110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:05:57.596276 18241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:57.597175 18241 log.cc:824] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:57.599735 18241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663: No bootstrap required, opened a new log
I20260810 23:05:57.602087 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:57.588040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39019 (local address 127.15.56.62:35511)
0810 23:05:57.588537 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:57.588553 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:57.588592 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:05:57.589469 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:57.589490 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:57.589858 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:57.590199 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:57.591037 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:57.592111 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:57.592984 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:57.593487 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:57.593636 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:57.596672 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:57.596679 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:57.596699 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:57.596735 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:57.598761 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:57.599238 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:57.599244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:57.599250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:57.599400 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:57.599645 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:57.599648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:57.599649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:57.600233 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:57.600444 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:57.600714 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 23:05:57.601051 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":157,"threads_started":1}
I20260810 23:05:57.604442 18241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5667be4d544258a7f2fa41fe139663" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35511 } }
I20260810 23:05:57.604709 18241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:57.604775 18241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5667be4d544258a7f2fa41fe139663, State: Initialized, Role: FOLLOWER
I20260810 23:05:57.605284 18241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5667be4d544258a7f2fa41fe139663" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35511 } }
I20260810 23:05:57.605415 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:57.605468 18241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:57.605538 18241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:57.605643 18241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:57.606739 18241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5667be4d544258a7f2fa41fe139663" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35511 } }
I20260810 23:05:57.607154 18241 leader_election.cc:302] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b5667be4d544258a7f2fa41fe139663; no voters:
I20260810 23:05:57.607563 18241 leader_election.cc:288] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:57.607887 18254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:57.608100 18254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [term 1 LEADER]: Becoming Leader. State: Replica: 1b5667be4d544258a7f2fa41fe139663, State: Running, Role: LEADER
I20260810 23:05:57.608544 18254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5667be4d544258a7f2fa41fe139663" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35511 } }
I20260810 23:05:57.609414 18241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:57.609915 18254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b5667be4d544258a7f2fa41fe139663. Latest consensus state: current_term: 1 leader_uuid: "1b5667be4d544258a7f2fa41fe139663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5667be4d544258a7f2fa41fe139663" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35511 } } }
I20260810 23:05:57.610113 18254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:57.610373 18264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b5667be4d544258a7f2fa41fe139663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5667be4d544258a7f2fa41fe139663" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 35511 } } }
I20260810 23:05:57.610483 18264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:57.610587 18268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:57.614307 18268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:57.618192 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:57.607727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53539 (local address 127.15.56.62:35511)
0810 23:05:57.607864 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:57.607870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:57.608289 (+ 419us) server_negotiation.cc:407] Connection header received
0810 23:05:57.608513 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:57.608517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:57.608576 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:57.608674 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:57.609611 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:57.610444 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:57.611752 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:57.612087 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:57.612179 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:57.614168 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:57.614172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:57.614175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:57.614207 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:57.616162 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:57.616767 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:57.616772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:57.616774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:57.616885 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:57.617215 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:57.617218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:57.617220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:57.617473 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:57.617605 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:57.617887 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 23:05:57.618030 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:05:57.621722 18268 catalog_manager.cc:1347] Generated new cluster ID: 11c7c4e010f9419b971a6c3e3d4c95ad
I20260810 23:05:57.621812 18268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:57.644096 18268 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:57.645140 18268 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:57.650615 18268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b5667be4d544258a7f2fa41fe139663: Generated new TSK 0
I20260810 23:05:57.651561 18268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:57.661172 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 18133
2026-08-10T23:05:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1583 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:57 dist-test-slave-6bbx krb5kdc[18289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:57 dist-test-slave-6bbx krb5kdc[18289](info): set up 2 sockets
Aug 10 23:05:57 dist-test-slave-6bbx krb5kdc[18289](info): commencing operation
krb5kdc: starting...
W20260810 23:05:58.824553 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:05:58 dist-test-slave-6bbx krb5kdc[18289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:05:58.956923 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:40089
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:40089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:59.074445 18311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:59.074767 18311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:59.074836 18311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:59.079365 18311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:59.079535 18311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:59.079578 18311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:59.079602 18311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:59.079628 18311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:59.085883 18311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:40089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:40089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:05:59.087044 18311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:59.088203 18311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:59.097169 18321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:59.097148 18322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:59.097142 18324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:59.097235 18311 server_base.cc:1034] running on GCE node
I20260810 23:05:59.098068 18311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:59.099313 18311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:59.100544 18311 hybrid_clock.cc:648] HybridClock initialized: now 1786403159100503 us; error 51 us; skew 500 ppm
Aug 10 23:05:59 dist-test-slave-6bbx krb5kdc[18289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403159, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:59.103732 18311 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:05:59.104768 18311 webserver.cc:460] Webserver started at http://127.15.56.62:36619/ using document root <none> and password file <none>
I20260810 23:05:59.105345 18311 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:59.105409 18311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:59.105592 18311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:59.107831 18311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "660c27de2e81459a8a5bfa5f33a84473"
format_stamp: "Formatted at 2026-08-10 23:05:59 on dist-test-slave-6bbx"
server_key: "00538b656168f7b25fd0d6bdb1125b8e"
server_key_iv: "8569c68c9dd85b3976c5878c5582362a"
server_key_version: "encryptionkey@0"
I20260810 23:05:59.108408 18311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "660c27de2e81459a8a5bfa5f33a84473"
format_stamp: "Formatted at 2026-08-10 23:05:59 on dist-test-slave-6bbx"
server_key: "00538b656168f7b25fd0d6bdb1125b8e"
server_key_iv: "8569c68c9dd85b3976c5878c5582362a"
server_key_version: "encryptionkey@0"
I20260810 23:05:59.112396 18311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:05:59.115010 18336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:59.116364 18311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:59.116482 18311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "660c27de2e81459a8a5bfa5f33a84473"
format_stamp: "Formatted at 2026-08-10 23:05:59 on dist-test-slave-6bbx"
server_key: "00538b656168f7b25fd0d6bdb1125b8e"
server_key_iv: "8569c68c9dd85b3976c5878c5582362a"
server_key_version: "encryptionkey@0"
I20260810 23:05:59.116600 18311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:59.171470 18311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:59.174932 18311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:59.175127 18311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:59.182432 18311 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:40089
I20260810 23:05:59.182463 18423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:40089 every 8 connection(s)
I20260810 23:05:59.183511 18311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:59.185199 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 18311
I20260810 23:05:59.185333 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:05:59.185621 15584 external_mini_cluster.cc:1428] Setting key 2a79a14f4b42dd9875fafc979b3871a4
I20260810 23:05:59.186960 18424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:59.194392 18424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473: Bootstrap starting.
I20260810 23:05:59.196097 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:59.187282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57287 (local address 127.15.56.62:40089)
0810 23:05:59.187842 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:59.187861 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:59.187903 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:05:59.188596 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:59.188627 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:59.189032 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:59.189369 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:59.189907 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:59.190993 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:59.192304 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:59.192834 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:59.192971 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:59.193979 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:59.193985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:59.194002 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:59.194425 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:59.194490 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:59.194515 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:59.194621 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:59.194772 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 23:05:59.195014 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":179,"threads_started":1}
I20260810 23:05:59.197439 18424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:59.198361 18424 log.cc:824] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:59.199533 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:59.200870 18424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473: No bootstrap required, opened a new log
I20260810 23:05:59.204205 18424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660c27de2e81459a8a5bfa5f33a84473" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40089 } }
I20260810 23:05:59.204457 18424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:59.204496 18424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660c27de2e81459a8a5bfa5f33a84473, State: Initialized, Role: FOLLOWER
I20260810 23:05:59.204918 18424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660c27de2e81459a8a5bfa5f33a84473" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40089 } }
I20260810 23:05:59.205044 18424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:59.205099 18424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:59.205199 18424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:59.206159 18424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660c27de2e81459a8a5bfa5f33a84473" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40089 } }
I20260810 23:05:59.206274 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:59.201985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35927 (local address 127.15.56.62:40089)
0810 23:05:59.202121 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:59.202126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:59.202344 (+ 218us) server_negotiation.cc:407] Connection header received
0810 23:05:59.202457 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:59.202460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:59.202506 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:59.202569 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:59.202833 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:59.203747 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:59.204566 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:59.204853 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:59.204914 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:59.205833 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:59.205837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:59.205840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:05:59.205853 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:59.205879 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:05:59.205890 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:59.205950 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:59.206039 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 23:05:59.206093 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:05:59.206570 18424 leader_election.cc:302] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 660c27de2e81459a8a5bfa5f33a84473; no voters:
I20260810 23:05:59.206929 18424 leader_election.cc:288] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:59.207010 18436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:59.207206 18436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [term 1 LEADER]: Becoming Leader. State: Replica: 660c27de2e81459a8a5bfa5f33a84473, State: Running, Role: LEADER
I20260810 23:05:59.207545 18436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660c27de2e81459a8a5bfa5f33a84473" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40089 } }
I20260810 23:05:59.208278 18424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:59.209298 18439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 660c27de2e81459a8a5bfa5f33a84473. Latest consensus state: current_term: 1 leader_uuid: "660c27de2e81459a8a5bfa5f33a84473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660c27de2e81459a8a5bfa5f33a84473" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40089 } } }
I20260810 23:05:59.209452 18439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:59.209290 18442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "660c27de2e81459a8a5bfa5f33a84473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "660c27de2e81459a8a5bfa5f33a84473" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 40089 } } }
I20260810 23:05:59.209683 18442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:59.209918 18450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:59.213049 18450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:59.220052 18450 catalog_manager.cc:1347] Generated new cluster ID: e7b82ab2d693470287c65cdb1f055359
I20260810 23:05:59.220132 18450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:59.232928 18450 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:59.233887 18450 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:59.240559 18450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 660c27de2e81459a8a5bfa5f33a84473: Generated new TSK 0
I20260810 23:05:59.241470 18450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:59.251700 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 18311
2026-08-10T23:05:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1590 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:59 dist-test-slave-6bbx krb5kdc[18464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:59 dist-test-slave-6bbx krb5kdc[18464](info): set up 2 sockets
Aug 10 23:05:59 dist-test-slave-6bbx krb5kdc[18464](info): commencing operation
krb5kdc: starting...
W20260810 23:06:00.445266 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:06:00 dist-test-slave-6bbx krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:06:00.576607 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43601
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:36493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:43601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:00.692605 18485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:00.692919 18485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:00.692961 18485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:00.697052 18485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:00.697137 18485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:00.697156 18485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:00.697171 18485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:00.697185 18485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:00.702713 18485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:36493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:43601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:00.703920 18485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:00.704977 18485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:00.713904 18493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:00.713948 18485 server_base.cc:1034] running on GCE node
W20260810 23:06:00.713888 18499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:00.713893 18495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:00.714700 18485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:00.715987 18485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:00.717149 18485 hybrid_clock.cc:648] HybridClock initialized: now 1786403160717131 us; error 31 us; skew 500 ppm
Aug 10 23:06:00 dist-test-slave-6bbx krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403160, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:00.720062 18485 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:06:00.721042 18485 webserver.cc:460] Webserver started at http://127.15.56.62:39115/ using document root <none> and password file <none>
I20260810 23:06:00.721612 18485 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:00.721674 18485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:00.721861 18485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:00.724126 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "85a21e41247b417fb241539626a697c3"
format_stamp: "Formatted at 2026-08-10 23:06:00 on dist-test-slave-6bbx"
server_key: "029c303efcca1e15fb5cbb83c58a0ae2"
server_key_iv: "6c3201876842d728a67cd25dcb844ebc"
server_key_version: "encryptionkey@0"
I20260810 23:06:00.724692 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "85a21e41247b417fb241539626a697c3"
format_stamp: "Formatted at 2026-08-10 23:06:00 on dist-test-slave-6bbx"
server_key: "029c303efcca1e15fb5cbb83c58a0ae2"
server_key_iv: "6c3201876842d728a67cd25dcb844ebc"
server_key_version: "encryptionkey@0"
I20260810 23:06:00.728826 18485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:06:00.731710 18514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:00.733168 18485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:06:00.733325 18485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "85a21e41247b417fb241539626a697c3"
format_stamp: "Formatted at 2026-08-10 23:06:00 on dist-test-slave-6bbx"
server_key: "029c303efcca1e15fb5cbb83c58a0ae2"
server_key_iv: "6c3201876842d728a67cd25dcb844ebc"
server_key_version: "encryptionkey@0"
I20260810 23:06:00.733464 18485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:00.739037 18485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:00.742316 18485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:00.742534 18485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:00.750392 18485 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:43601
I20260810 23:06:00.750422 18597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:43601 every 8 connection(s)
I20260810 23:06:00.751427 18485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:00.754360 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 18485
I20260810 23:06:00.754487 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:06:00.754561 18598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:00.754791 15584 external_mini_cluster.cc:1428] Setting key 28b61a14d6e0343fd17691a9efa020c8
I20260810 23:06:00.760718 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:00.756341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49387 (local address 127.15.56.62:43601)
0810 23:06:00.756817 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:00.756838 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:00.756885 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:06:00.757638 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:00.757664 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:00.758068 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:00.758473 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:00.759165 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:00.759171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:00.759186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:00.759197 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:00.759260 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:00.759287 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:00.759429 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:00.759502 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 23:06:00.759709 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":122,"threads_started":1}
I20260810 23:06:00.762499 18598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3: Bootstrap starting.
I20260810 23:06:00.763890 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:06:00.765410 18598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:00.766356 18598 log.cc:824] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:00.768247 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:00.766415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36153 (local address 127.15.56.62:43601)
0810 23:06:00.766542 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:00.766546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:00.766806 (+ 260us) server_negotiation.cc:407] Connection header received
0810 23:06:00.766925 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:00.766928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:00.766959 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:00.767024 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:00.767885 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:00.767889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:00.767892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:06:00.767904 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:00.767935 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:06:00.767948 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:00.768042 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:00.768086 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 23:06:00.768124 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:00.768925 18598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3: No bootstrap required, opened a new log
I20260810 23:06:00.772893 18598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a21e41247b417fb241539626a697c3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43601 } }
I20260810 23:06:00.773247 18598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:00.773324 18598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a21e41247b417fb241539626a697c3, State: Initialized, Role: FOLLOWER
I20260810 23:06:00.773802 18598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a21e41247b417fb241539626a697c3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43601 } }
I20260810 23:06:00.773941 18598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:00.773990 18598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:00.774068 18598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:00.775028 18598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a21e41247b417fb241539626a697c3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43601 } }
I20260810 23:06:00.775368 18598 leader_election.cc:302] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85a21e41247b417fb241539626a697c3; no voters:
I20260810 23:06:00.775681 18598 leader_election.cc:288] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:00.775867 18620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:00.776077 18620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [term 1 LEADER]: Becoming Leader. State: Replica: 85a21e41247b417fb241539626a697c3, State: Running, Role: LEADER
I20260810 23:06:00.776434 18620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a21e41247b417fb241539626a697c3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43601 } }
I20260810 23:06:00.776998 18598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:00.778211 18623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85a21e41247b417fb241539626a697c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a21e41247b417fb241539626a697c3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43601 } } }
I20260810 23:06:00.778331 18623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:00.778568 18625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a21e41247b417fb241539626a697c3. Latest consensus state: current_term: 1 leader_uuid: "85a21e41247b417fb241539626a697c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a21e41247b417fb241539626a697c3" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43601 } } }
I20260810 23:06:00.778643 18625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:00.778939 18635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:00.781960 18635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:00.788170 18635 catalog_manager.cc:1347] Generated new cluster ID: 8ff3090bb50b4bbd90d6ade12515797f
I20260810 23:06:00.788264 18635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:00.801975 18635 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:00.802366 18635 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:00.807194 18635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85a21e41247b417fb241539626a697c3: Generated new TSK 0
I20260810 23:06:00.808043 18635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:06:00.821684 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 18485
2026-08-10T23:06:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1562 ms)
[----------] 18 tests from AuthTokenIssuingTest (28257 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:00 dist-test-slave-6bbx krb5kdc[18651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:00 dist-test-slave-6bbx krb5kdc[18651](info): set up 2 sockets
Aug 10 23:06:00 dist-test-slave-6bbx krb5kdc[18651](info): commencing operation
krb5kdc: starting...
W20260810 23:06:01.990108 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:06:02.123127 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:46615
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:46615,127.15.56.61:33269,127.15.56.60:34045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:02.243199 18668 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:06:02.243507 18668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:02.243530 18668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:02.243567 18668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:02.247732 18668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:02.247802 18668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:02.247821 18668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:02.247836 18668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:02.247850 18668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:02.253334 18668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:46615,127.15.56.61:33269,127.15.56.60:34045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:02.254464 18668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:02.255537 18668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:02.264394 18681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:02.264495 18668 server_base.cc:1034] running on GCE node
W20260810 23:06:02.264394 18690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:02.264634 18683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:02.265264 18668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:02.266491 18668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:02.267695 18668 hybrid_clock.cc:648] HybridClock initialized: now 1786403162267652 us; error 42 us; skew 500 ppm
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:02.270777 18668 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:06:02.271847 18668 webserver.cc:460] Webserver started at http://127.15.56.62:43451/ using document root <none> and password file <none>
I20260810 23:06:02.272444 18668 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:02.272518 18668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:02.272748 18668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:02.275072 18668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "bbebfaccb68b4f25af77d492c063ef8e"
format_stamp: "Formatted at 2026-08-10 23:06:02 on dist-test-slave-6bbx"
server_key: "8ed45a2918a5c1662d12be2f425c31b0"
server_key_iv: "1e102bfc34953ed4b4485df0bedfdd2e"
server_key_version: "encryptionkey@0"
I20260810 23:06:02.275688 18668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "bbebfaccb68b4f25af77d492c063ef8e"
format_stamp: "Formatted at 2026-08-10 23:06:02 on dist-test-slave-6bbx"
server_key: "8ed45a2918a5c1662d12be2f425c31b0"
server_key_iv: "1e102bfc34953ed4b4485df0bedfdd2e"
server_key_version: "encryptionkey@0"
I20260810 23:06:02.279713 18668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:06:02.282374 18702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:02.283876 18668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:06:02.284019 18668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "bbebfaccb68b4f25af77d492c063ef8e"
format_stamp: "Formatted at 2026-08-10 23:06:02 on dist-test-slave-6bbx"
server_key: "8ed45a2918a5c1662d12be2f425c31b0"
server_key_iv: "1e102bfc34953ed4b4485df0bedfdd2e"
server_key_version: "encryptionkey@0"
I20260810 23:06:02.284134 18668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:02.308455 18668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:02.311992 18668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:02.312214 18668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:02.319723 18668 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:46615
I20260810 23:06:02.320026 18801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:46615 every 8 connection(s)
I20260810 23:06:02.320839 18668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:02.321638 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 18668
I20260810 23:06:02.321776 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:06:02.322089 15584 external_mini_cluster.cc:1428] Setting key a4fe7003328feb4c0738940568761b9a
I20260810 23:06:02.324712 18803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:02.329234 18803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.56.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.61@KRBTEST.COM" created.
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:06:02.345712 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.332204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39548 (local address 127.15.56.62:46615)
0810 23:06:02.332758 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.332789 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.333069 (+ 280us) server_negotiation.cc:407] Connection header received
0810 23:06:02.334179 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.334185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.334451 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.334669 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.335485 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.336429 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.337509 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.337952 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.338074 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.340776 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.340782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.340794 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.340817 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.342458 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.342995 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.343000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.343002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.343107 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.343398 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.343402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.343403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.343844 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.343969 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.344330 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.344577 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":163,"threads_started":1}
I20260810 23:06:02.345712 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.332207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:46615 (local address 127.0.0.1:39548)
0810 23:06:02.332891 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.332943 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:02.333836 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:02.334661 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:02.334666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:02.335041 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:02.335324 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.335339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.336571 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.336575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.337398 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.337408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.338119 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.338123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.338255 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.338844 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:02.338865 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:02.340605 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:02.342571 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.342581 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.342593 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.342897 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.343193 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.343196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.343198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.343314 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.343949 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:02.343956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:02.344092 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 23:06:02.344251 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 23:06:02.344636 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":68,"spinlock_wait_cycles":62848,"thread_start_us":163,"threads_started":1}
I20260810 23:06:02.348309 18803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:02.349088 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.348795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:33269 (local address 127.0.0.1:44360)
0810 23:06:02.348933 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.349013 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:06:02.349344 18704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.56.61:33269: connect: Connection refused (error 111)
I20260810 23:06:02.350297 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.350056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:33269 (local address 127.0.0.1:44362)
0810 23:06:02.350190 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.350237 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 23:06:02.350481 18803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.61:33269: Network error: Client connection negotiation failed: client connection to 127.15.56.61:33269: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab.
Entry for principal kudu/127.15.56.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab.
I20260810 23:06:02.354902 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.61:33269
--webserver_interface=127.15.56.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab
--principal=kudu/127.15.56.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:46615,127.15.56.61:33269,127.15.56.60:34045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:02.399858 18803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } attempt: 1
I20260810 23:06:02.400791 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.400463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:33269 (local address 127.0.0.1:44370)
0810 23:06:02.400630 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.400700 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 23:06:02.401628 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.401379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:33269 (local address 127.0.0.1:44380)
0810 23:06:02.401503 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.401553 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:06:02.401762 18803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.61:33269: Network error: Client connection negotiation failed: client connection to 127.15.56.61:33269: connect: Connection refused (error 111)
I20260810 23:06:02.469949 18803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } attempt: 2
I20260810 23:06:02.470914 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.470581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:33269 (local address 127.0.0.1:44396)
0810 23:06:02.470752 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.470820 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 23:06:02.471742 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.471470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:33269 (local address 127.0.0.1:44398)
0810 23:06:02.471581 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.471632 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 23:06:02.471872 18803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.61:33269: Network error: Client connection negotiation failed: client connection to 127.15.56.61:33269: connect: Connection refused (error 111)
W20260810 23:06:02.475154 18812 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:06:02.475479 18812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:02.475519 18812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:02.475581 18812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:02.479756 18812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:02.479820 18812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:02.479837 18812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:02.479856 18812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:02.479884 18812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:02.485518 18812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:46615,127.15.56.61:33269,127.15.56.60:34045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab
--principal=kudu/127.15.56.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.61:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.56.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:02.486732 18812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:02.487900 18812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:02.497560 18831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:02.497565 18821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:02.497560 18824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:02.497614 18812 server_base.cc:1034] running on GCE node
I20260810 23:06:02.498404 18812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:02.499627 18812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:02.500842 18812 hybrid_clock.cc:648] HybridClock initialized: now 1786403162500820 us; error 32 us; skew 500 ppm
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:02.503986 18812 init.cc:377] Logged in from keytab as kudu/127.15.56.61@KRBTEST.COM (short username kudu)
I20260810 23:06:02.504976 18812 webserver.cc:460] Webserver started at http://127.15.56.61:38165/ using document root <none> and password file <none>
I20260810 23:06:02.505570 18812 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:02.505640 18812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:02.505859 18812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:02.508054 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/data/instance:
uuid: "d394f0d9b5dc40018697da603cd7614e"
format_stamp: "Formatted at 2026-08-10 23:06:02 on dist-test-slave-6bbx"
server_key: "614cd563042c3529183456e44f662c92"
server_key_iv: "b64f7c4c4c6654777ac7b0145eeeec39"
server_key_version: "encryptionkey@0"
I20260810 23:06:02.508616 18812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/wal/instance:
uuid: "d394f0d9b5dc40018697da603cd7614e"
format_stamp: "Formatted at 2026-08-10 23:06:02 on dist-test-slave-6bbx"
server_key: "614cd563042c3529183456e44f662c92"
server_key_iv: "b64f7c4c4c6654777ac7b0145eeeec39"
server_key_version: "encryptionkey@0"
I20260810 23:06:02.512508 18812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 23:06:02.515147 18840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:02.516438 18812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:02.516557 18812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/wal
uuid: "d394f0d9b5dc40018697da603cd7614e"
format_stamp: "Formatted at 2026-08-10 23:06:02 on dist-test-slave-6bbx"
server_key: "614cd563042c3529183456e44f662c92"
server_key_iv: "b64f7c4c4c6654777ac7b0145eeeec39"
server_key_version: "encryptionkey@0"
I20260810 23:06:02.516656 18812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:02.533785 18812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:02.536907 18812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:02.537101 18812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:02.544329 18812 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.61:33269
I20260810 23:06:02.544677 18926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.61:33269 every 8 connection(s)
I20260810 23:06:02.545388 18812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
I20260810 23:06:02.548552 18927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:02.552896 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 18812
I20260810 23:06:02.553030 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-1/wal/instance
I20260810 23:06:02.552721 18927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:02.553319 15584 external_mini_cluster.cc:1428] Setting key 4b66ff492e061f03321e7cce654c06b8
WARNING: no policy specified for kudu/127.15.56.60@KRBTEST.COM; defaulting to no policy
I20260810 23:06:02.563012 18803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.60@KRBTEST.COM" created.
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:06:02.569785 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.555242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.15.56.62:46615)
0810 23:06:02.555407 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.555411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.556514 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 23:06:02.557498 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.557503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.557564 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.557637 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.559007 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.559555 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.560722 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.560964 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.561063 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.565420 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.565424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.565426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.565450 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.567202 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.567918 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.567922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.567924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.568023 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.568437 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.568441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.568443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.568649 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.568739 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.569473 (+ 734us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.569607 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
I20260810 23:06:02.571751 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.555782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:46615 (local address 127.0.0.1:39564)
0810 23:06:02.556350 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.556398 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:02.557284 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:02.557845 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:02.557852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:02.558232 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:02.558854 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.558870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.559701 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.559705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.560599 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.560608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.561120 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.561124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.561291 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.561879 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:02.561903 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:02.565266 (+ 3363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:02.567309 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.567317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.567339 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.567790 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.568128 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.568131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.568133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.568330 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.568760 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:02.568767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:02.569129 (+ 362us) client_negotiation.cc:769] Sending connection context
0810 23:06:02.570045 (+ 916us) client_negotiation.cc:240] Negotiation successful
0810 23:06:02.570438 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":175,"threads_started":1}
I20260810 23:06:02.573504 18927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:02.574980 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.563470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:33269 (local address 127.0.0.1:44408)
0810 23:06:02.563691 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.563707 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:02.563789 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:02.564418 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:02.564421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:02.564612 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:02.564750 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.564758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.566127 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.566131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.567030 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.567045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.567874 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.567878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.567999 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.568787 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:02.568806 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:02.570741 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:02.573066 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.573069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.573071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.573327 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.573718 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.573722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.573724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.573821 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.574380 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:02.574384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:02.574584 (+ 200us) client_negotiation.cc:769] Sending connection context
0810 23:06:02.574698 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 23:06:02.574852 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 23:06:02.575171 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.563612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.15.56.61:33269)
0810 23:06:02.563961 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.563971 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.563994 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:06:02.564032 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.564037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.564214 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.564455 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.564933 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.565976 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.567200 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.567613 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.567858 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.570898 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.570910 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.570928 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.570956 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.572942 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.573470 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.573477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.573478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.573592 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.573938 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.573940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.573942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.574272 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.574417 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.574821 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.574982 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":148,"threads_started":1}
I20260810 23:06:02.577209 18803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:02.577935 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.577679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:36926)
0810 23:06:02.577808 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.577862 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 23:06:02.578701 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.578423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:36940)
0810 23:06:02.578582 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.578629 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260810 23:06:02.578876 18803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:34045: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab.
Entry for principal kudu/127.15.56.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab.
I20260810 23:06:02.584002 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.574042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:33269 (local address 127.0.0.1:44418)
0810 23:06:02.574154 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.574168 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:02.574246 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:02.575761 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:02.575764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:02.575905 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:02.576007 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.576013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.577065 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.577069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.577718 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.577728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.578303 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.578307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.578445 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.578965 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:02.578980 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:02.580251 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:02.582383 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.582386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.582387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.582634 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.582896 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.582899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.582900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.582996 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.583423 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:02.583425 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:02.583524 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 23:06:02.583648 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 23:06:02.583832 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":20}
I20260810 23:06:02.584072 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.574180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44418 (local address 127.15.56.61:33269)
0810 23:06:02.574525 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.574535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.574547 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:06:02.574580 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.574582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.574622 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.574703 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.576130 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.576929 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.577828 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.578178 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.578292 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.580409 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.580413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.580415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.580437 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.582253 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.582728 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.582730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.582731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.582808 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.583078 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.583081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.583082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.583284 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.583393 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.583728 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.583908 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":282,"thread_start_us":120,"threads_started":1}
I20260810 23:06:02.584749 18927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:02.585242 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.60:34045
--webserver_interface=127.15.56.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab
--principal=kudu/127.15.56.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:41151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:46615,127.15.56.61:33269,127.15.56.60:34045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:02.585525 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.585235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:36956)
0810 23:06:02.585362 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.585462 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:06:02.585764 18841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
I20260810 23:06:02.586612 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.586393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:36964)
0810 23:06:02.586503 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.586555 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:06:02.586750 18927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:34045: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
I20260810 23:06:02.610085 18803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } attempt: 1
I20260810 23:06:02.610989 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.610688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:36976)
0810 23:06:02.610839 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.610903 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:06:02.611865 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.611558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:36992)
0810 23:06:02.611731 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.611787 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 23:06:02.612042 18803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:34045: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
I20260810 23:06:02.636065 18927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } attempt: 1
I20260810 23:06:02.637039 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.636688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:37004)
0810 23:06:02.636878 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.636947 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:06:02.637836 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.637599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:37008)
0810 23:06:02.637726 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.637773 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 23:06:02.637955 18927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:34045: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
I20260810 23:06:02.687232 18803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } attempt: 2
I20260810 23:06:02.688149 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.687838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:37022)
0810 23:06:02.687993 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.688061 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:06:02.688899 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.688669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:37030)
0810 23:06:02.688786 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.688842 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 23:06:02.689064 18803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:34045: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
W20260810 23:06:02.702674 18951 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:06:02.702980 18951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:02.703006 18951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:02.703066 18951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:06:02.706099 18927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } attempt: 2
I20260810 23:06:02.706950 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.706643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:37040)
0810 23:06:02.706796 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.706865 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 23:06:02.707360 18951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:02.707420 18951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:02.707448 18951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:02.707468 18951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:02.707489 18951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:02.707929 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.707618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:37046)
0810 23:06:02.707799 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.707865 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260810 23:06:02.708060 18927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:34045: Network error: Client connection negotiation failed: client connection to 127.15.56.60:34045: connect: Connection refused (error 111)
I20260810 23:06:02.713428 18951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:41151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:46615,127.15.56.61:33269,127.15.56.60:34045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab
--principal=kudu/127.15.56.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.60:34045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.56.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:02.714685 18951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:02.715839 18951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:02.724911 18959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:02.725131 18951 server_base.cc:1034] running on GCE node
W20260810 23:06:02.724897 18960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:02.724900 18966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:02.725886 18951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:02.727350 18951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:02.728578 18951 hybrid_clock.cc:648] HybridClock initialized: now 1786403162728548 us; error 48 us; skew 500 ppm
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:02.731817 18951 init.cc:377] Logged in from keytab as kudu/127.15.56.60@KRBTEST.COM (short username kudu)
I20260810 23:06:02.732906 18951 webserver.cc:460] Webserver started at http://127.15.56.60:42995/ using document root <none> and password file <none>
I20260810 23:06:02.733505 18951 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:02.733570 18951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:02.733812 18951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:02.736128 18951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/data/instance:
uuid: "3f7eaedb09434e18b44a92efc80487c6"
format_stamp: "Formatted at 2026-08-10 23:06:02 on dist-test-slave-6bbx"
server_key: "f29b2d78c114af23d6bd889bb203efff"
server_key_iv: "d4faa007e0517ebfb052370858b955ea"
server_key_version: "encryptionkey@0"
I20260810 23:06:02.736778 18951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/wal/instance:
uuid: "3f7eaedb09434e18b44a92efc80487c6"
format_stamp: "Formatted at 2026-08-10 23:06:02 on dist-test-slave-6bbx"
server_key: "f29b2d78c114af23d6bd889bb203efff"
server_key_iv: "d4faa007e0517ebfb052370858b955ea"
server_key_version: "encryptionkey@0"
I20260810 23:06:02.741210 18951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:06:02.743914 18977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:02.745325 18951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:06:02.745442 18951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/wal
uuid: "3f7eaedb09434e18b44a92efc80487c6"
format_stamp: "Formatted at 2026-08-10 23:06:02 on dist-test-slave-6bbx"
server_key: "f29b2d78c114af23d6bd889bb203efff"
server_key_iv: "d4faa007e0517ebfb052370858b955ea"
server_key_version: "encryptionkey@0"
I20260810 23:06:02.745568 18951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:02.759538 18951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:02.763058 18951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:02.763260 18951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:02.770628 18951 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.60:34045
I20260810 23:06:02.770907 19058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.60:34045 every 8 connection(s)
I20260810 23:06:02.771725 18951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
I20260810 23:06:02.772982 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 18951
I20260810 23:06:02.773113 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402867637343-15584-0/minicluster-data/master-2/wal/instance
I20260810 23:06:02.773423 15584 external_mini_cluster.cc:1428] Setting key d8b10752eb3e8509fc97a2b19829c5d5
I20260810 23:06:02.775332 19059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:02.779409 19059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:06:02.788245 18803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } attempt: 3
I20260810 23:06:02.788729 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.774972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.15.56.60:34045)
0810 23:06:02.775449 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.775472 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.775515 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:06:02.776240 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.776259 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.776621 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.776953 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.777805 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.778881 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.779808 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.780365 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.780521 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.783335 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.783341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.783351 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.783380 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.785065 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.785508 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.785513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.785519 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.785642 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.786042 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.786045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.786046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.786654 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.786878 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.787227 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.787570 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":143,"threads_started":1}
I20260810 23:06:02.790997 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.781360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39580 (local address 127.15.56.62:46615)
0810 23:06:02.781490 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.781493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.782027 (+ 534us) server_negotiation.cc:407] Connection header received
0810 23:06:02.782189 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.782192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.782250 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.782319 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.783094 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.783593 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.784509 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.784730 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.784823 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.787331 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.787335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.787337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.787378 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.789028 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.789554 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.789557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.789559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.789673 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.789977 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.789980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.789982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.790239 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.790335 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.790698 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.790830 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:06:02.791018 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.781313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:46615 (local address 127.0.0.1:39580)
0810 23:06:02.781879 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.781938 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:02.782080 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:02.782389 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:02.782396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:02.782809 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:02.782968 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.782975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.783743 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.783746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.784387 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.784397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.784835 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.784839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.784962 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.785456 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:02.785470 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:02.787154 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:02.789142 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.789149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.789154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.789465 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.789784 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.789787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.789789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.789892 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.790353 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:02.790360 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:02.790480 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 23:06:02.790612 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 23:06:02.790833 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":199,"threads_started":1}
I20260810 23:06:02.792850 19059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:06:02.798661 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.788739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:37060)
0810 23:06:02.788884 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.788901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:02.788989 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:02.789512 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:02.789515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:02.789668 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:02.789782 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.789789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.790822 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.790826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.791564 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.791575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.792217 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.792221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.792310 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.792852 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:02.792866 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:02.794428 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:02.797202 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.797205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.797208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.797407 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.797666 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.797669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.797671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.797756 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.798142 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:02.798145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:02.798247 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 23:06:02.798334 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:06:02.798479 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 23:06:02.798661 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.789134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.15.56.60:34045)
0810 23:06:02.789260 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.789264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.789286 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:06:02.789319 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.789321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.789365 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.789893 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.789981 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.790689 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.791799 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.792107 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.792202 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.794560 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.794564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.794567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.794591 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.797109 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.797495 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.797499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.797501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.797582 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.797838 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.797841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.797842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.798051 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.798148 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.798397 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.798502 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:06:02.799932 18927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } attempt: 3
I20260810 23:06:02.802461 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.793572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.15.56.61:33269)
0810 23:06:02.793704 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.793708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.793723 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:06:02.793768 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.793771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.793820 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.793913 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.794358 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.795041 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.795835 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.796080 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.796169 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.798769 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.798773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.798776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.798807 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.800552 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.801088 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.801091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.801093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.801182 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.801506 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.801510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.801512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.801720 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.801793 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.802161 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.802300 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:06:02.802898 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.792242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.15.56.62:46615)
0810 23:06:02.792351 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.792355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.792876 (+ 521us) server_negotiation.cc:407] Connection header received
0810 23:06:02.793062 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.793064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.793111 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.793177 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.794284 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.794887 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.795796 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.796114 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.796679 (+ 565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.799255 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.799259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.799261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.799285 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.800718 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.801175 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.801178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.801180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.801278 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.801573 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.801576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.801577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.801764 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.801961 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.802633 (+ 672us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.802749 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 23:06:02.803223 18803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e: Bootstrap starting.
I20260810 23:06:02.803382 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.793413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:33269 (local address 127.0.0.1:44428)
0810 23:06:02.793522 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.793535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:02.793601 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:02.793942 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:02.793945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:02.794103 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:02.794211 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.794219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.795157 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.795160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.795702 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.795712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.796186 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.796190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.796293 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.796671 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:02.796686 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:02.798628 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:02.800733 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.800736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.800738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.800973 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.801310 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.801312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.801314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.801403 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.801873 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:02.801875 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:02.801963 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 23:06:02.802847 (+ 884us) client_negotiation.cc:240] Negotiation successful
0810 23:06:02.803171 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 23:06:02.804359 19059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
I20260810 23:06:02.807072 18803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:02.808012 18803 log.cc:824] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:02.810544 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.800359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:37074)
0810 23:06:02.800607 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.800622 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:02.800701 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:02.801179 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:02.801182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:02.801348 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:02.801448 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.801454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.802458 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.802462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.803205 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.803217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.804380 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.804384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.804474 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.804928 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:02.804942 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:02.806385 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:02.808912 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.808916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.808918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.809147 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.809434 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.809437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.809439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.809532 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.810046 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:02.810049 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:02.810148 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 23:06:02.810233 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:06:02.810376 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
I20260810 23:06:02.811154 18803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e: No bootstrap required, opened a new log
I20260810 23:06:02.811545 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.800846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.15.56.60:34045)
0810 23:06:02.800958 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.800962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.800974 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:06:02.801006 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.801009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.801046 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.801557 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.801649 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.802327 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.803358 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.804249 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.804683 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.806515 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.806519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.806521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.806544 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.808794 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.809259 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.809263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.809264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.809343 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.809635 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.809638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.809640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.809927 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.811060 (+ 1133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.811143 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.811392 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":896}
Aug 10 23:06:02 dist-test-slave-6bbx krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
I20260810 23:06:02.814657 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.804817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:34045 (local address 127.0.0.1:37088)
0810 23:06:02.805047 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.805056 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:02.805108 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:02.806261 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:02.806263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:02.806392 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:02.806451 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.806457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.807420 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.807424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.808188 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:02.808200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.808789 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.808793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:02.808877 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.809637 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:02.809651 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:02.811002 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:02.813057 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.813060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.813062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.813281 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.813539 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:02.813542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:02.813544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:02.813633 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:02.814170 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:02.814173 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:02.814257 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 23:06:02.814347 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:06:02.814484 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260810 23:06:02.815232 18927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e: Bootstrap starting.
I20260810 23:06:02.815281 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.805614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.15.56.60:34045)
0810 23:06:02.805985 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.806003 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.806017 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:06:02.806050 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.806052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.806091 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.806172 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.806552 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.807297 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.808304 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.808660 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.808779 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.811147 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.811150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.811152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.811176 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.812944 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.813377 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.813380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.813382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.813454 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.813717 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.813719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.813721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.814048 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.814137 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.814966 (+ 829us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.815087 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":130,"threads_started":1}
I20260810 23:06:02.815825 18803 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } }
I20260810 23:06:02.816082 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.804067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.15.56.61:33269)
0810 23:06:02.804713 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.804717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.804730 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:06:02.804761 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.804764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.804804 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.804877 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.807003 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.807602 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.809105 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.809392 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.809496 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.812174 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.812178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.812180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.812207 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.813877 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.814354 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.814357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.814359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.814443 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.814728 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.814731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.814733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.814947 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.815099 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.815829 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.815945 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572}
I20260810 23:06:02.816089 18803 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbebfaccb68b4f25af77d492c063ef8e, State: Initialized, Role: FOLLOWER
I20260810 23:06:02.816494 18803 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } }
I20260810 23:06:02.817252 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } } }
I20260810 23:06:02.817822 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:02.818460 18803 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:02.818535 19059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6: Bootstrap starting.
I20260810 23:06:02.819682 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:06:02.821549 18751 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bbebfaccb68b4f25af77d492c063ef8e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39590
I20260810 23:06:02.821781 18751 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:06:02.821904 18751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:02.823287 18751 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } }
I20260810 23:06:02.824326 18751 leader_election.cc:288] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [CANDIDATE]: Term 1 election: Requested vote from peers d394f0d9b5dc40018697da603cd7614e (127.15.56.61:33269), 3f7eaedb09434e18b44a92efc80487c6 (127.15.56.60:34045)
W20260810 23:06:02.824453 19090 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:02.824517 19090 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:02.828193 18927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:02.829229 18927 log.cc:824] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:02.830577 19059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:02.828192 18668 master_runner.cc:186] Error getting master registration for 127.15.56.61:33269: OK, instance_id { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" instance_seqno: 1786403162542250 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:06:02.831478 19059 log.cc:824] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:02.831645 18668 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:06:02.833364 18927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e: No bootstrap required, opened a new log
I20260810 23:06:02.833696 19059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6: No bootstrap required, opened a new log
I20260810 23:06:02.837657 18927 raft_consensus.cc:348] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } }
I20260810 23:06:02.837659 19059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } }
I20260810 23:06:02.837941 18927 raft_consensus.cc:729] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d394f0d9b5dc40018697da603cd7614e, State: Initialized, Role: FOLLOWER
I20260810 23:06:02.837932 19059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7eaedb09434e18b44a92efc80487c6, State: Initialized, Role: FOLLOWER
I20260810 23:06:02.838315 19059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } }
I20260810 23:06:02.838315 18927 consensus_queue.cc:260] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } }
I20260810 23:06:02.839092 19114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } } }
I20260810 23:06:02.839471 19114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:02.839944 18927 sys_catalog.cc:564] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:02.840034 18881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbebfaccb68b4f25af77d492c063ef8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d394f0d9b5dc40018697da603cd7614e"
I20260810 23:06:02.840283 18881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:02.842374 18881 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbebfaccb68b4f25af77d492c063ef8e in term 1.
I20260810 23:06:02.843114 18704 leader_election.cc:302] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbebfaccb68b4f25af77d492c063ef8e, d394f0d9b5dc40018697da603cd7614e; no voters:
I20260810 23:06:02.843492 19073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:02.844664 19073 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [term 1 LEADER]: Becoming Leader. State: Replica: bbebfaccb68b4f25af77d492c063ef8e, State: Running, Role: LEADER
I20260810 23:06:02.844780 19059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:02.845077 19073 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } }
I20260810 23:06:02.847271 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbebfaccb68b4f25af77d492c063ef8e. Latest consensus state: current_term: 1 leader_uuid: "bbebfaccb68b4f25af77d492c063ef8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } } }
I20260810 23:06:02.847398 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:02.848105 19024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbebfaccb68b4f25af77d492c063ef8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3f7eaedb09434e18b44a92efc80487c6"
I20260810 23:06:02.849460 19024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:02.851054 19024 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbebfaccb68b4f25af77d492c063ef8e in term 1.
I20260810 23:06:02.853272 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.827651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.15.56.60:34045)
0810 23:06:02.827895 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.827899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.843986 (+ 16087us) server_negotiation.cc:407] Connection header received
0810 23:06:02.844226 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.844230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.844289 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.844344 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.845353 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.846107 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.847439 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.847797 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.847879 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.849634 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.849638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.849641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.849669 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.851270 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.851798 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.851803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.851804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.851899 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.852245 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.852249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.852250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.852485 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.852591 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.852986 (+ 395us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.853116 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":175}
I20260810 23:06:02.855830 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.826458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.15.56.62:46615)
0810 23:06:02.826567 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.826572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.827399 (+ 827us) server_negotiation.cc:407] Connection header received
0810 23:06:02.827789 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.827793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.827846 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.827898 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.829224 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.830009 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.831102 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.831430 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.831537 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.843218 (+ 11681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.843222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.843225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.843259 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.852184 (+ 8925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.852692 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.852696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.852698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.852788 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.853060 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.853064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.853065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.853274 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.853390 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.855487 (+ 2097us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.855632 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 23:06:02.857247 19115 sys_catalog.cc:455] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } } }
I20260810 23:06:02.857465 19115 sys_catalog.cc:458] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:02.859426 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.827526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.15.56.61:33269)
0810 23:06:02.827805 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.827810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.840073 (+ 12263us) server_negotiation.cc:407] Connection header received
0810 23:06:02.840318 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.840322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.840380 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.840436 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.841592 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.842358 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.843483 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:02.843791 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:02.843878 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:02.846747 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:02.846752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:02.846755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:02.846785 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:02.848455 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.857700 (+ 9245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.857705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.857707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.857814 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:02.858074 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:02.858078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:02.858079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:02.858320 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:02.858466 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:02.859146 (+ 680us) server_negotiation.cc:299] Negotiation successful
0810 23:06:02.859271 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260810 23:06:02.859791 19156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:02.860821 19156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 23:06:02.861766 19132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:02.861842 19132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:06:02.866793 19155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 23:06:02.867354 18881 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [term 1 FOLLOWER]: Refusing update from remote peer bbebfaccb68b4f25af77d492c063ef8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:02.868232 19133 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:02.867393 19024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [term 1 FOLLOWER]: Refusing update from remote peer bbebfaccb68b4f25af77d492c063ef8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260810 23:06:02.872583 19155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:02.872582 19115 sys_catalog.cc:455] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbebfaccb68b4f25af77d492c063ef8e. Latest consensus state: current_term: 1 leader_uuid: "bbebfaccb68b4f25af77d492c063ef8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } } }
I20260810 23:06:02.872733 19115 sys_catalog.cc:458] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:02.874185 19133 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:02.878473 19114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbebfaccb68b4f25af77d492c063ef8e. Latest consensus state: current_term: 1 leader_uuid: "bbebfaccb68b4f25af77d492c063ef8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } } }
I20260810 23:06:02.878613 19114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:02.881563 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbebfaccb68b4f25af77d492c063ef8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } } }
I20260810 23:06:02.881719 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:02.882941 19158 mvcc.cc:205] Tried to move back new op lower bound from 7317107355097636864 to 7317107355017752576. Current Snapshot: MvccSnapshot[applied={T|T < 7317107355097636864}]
I20260810 23:06:02.883518 19115 sys_catalog.cc:455] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbebfaccb68b4f25af77d492c063ef8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } } }
I20260810 23:06:02.883643 19115 sys_catalog.cc:458] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:02.885365 19160 mvcc.cc:205] Tried to move back new op lower bound from 7317107355097636864 to 7317107355017752576. Current Snapshot: MvccSnapshot[applied={T|T < 7317107355097636864}]
I20260810 23:06:02.890962 19133 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bbebfaccb68b4f25af77d492c063ef8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } } }
I20260810 23:06:02.891105 19133 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:02.892373 19114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bbebfaccb68b4f25af77d492c063ef8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbebfaccb68b4f25af77d492c063ef8e" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 46615 } } peers { permanent_uuid: "d394f0d9b5dc40018697da603cd7614e" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 33269 } } peers { permanent_uuid: "3f7eaedb09434e18b44a92efc80487c6" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 34045 } } }
I20260810 23:06:02.892503 19114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:02.893919 19162 mvcc.cc:205] Tried to move back new op lower bound from 7317107355097636864 to 7317107355017752576. Current Snapshot: MvccSnapshot[applied={T|T < 7317107355097636864}]
I20260810 23:06:02.899874 19156 catalog_manager.cc:1347] Generated new cluster ID: c870f4392c074aae8bda97590ccd8c72
I20260810 23:06:02.899958 19156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:02.917379 19156 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:02.918787 19156 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:02.939999 19156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbebfaccb68b4f25af77d492c063ef8e: Generated new TSK 0
I20260810 23:06:02.940965 19156 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:06:02.973860 18807 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:02.970791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.15.56.62:46615)
0810 23:06:02.970927 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.970932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.972701 (+ 1769us) server_negotiation.cc:407] Connection header received
0810 23:06:02.972927 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.972932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.972985 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.973040 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.973693 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39604: BlockingRecv error: recv got EOF from 127.0.0.1:39604 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 23:06:02.975455 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:02.971568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.15.56.61:33269)
0810 23:06:02.971710 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.971715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.974365 (+ 2650us) server_negotiation.cc:407] Connection header received
0810 23:06:02.974540 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.974544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.974595 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.974644 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.975325 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44450: BlockingRecv error: recv got EOF from 127.0.0.1:44450 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 23:06:02.977404 19071 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:02.972190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.15.56.60:34045)
0810 23:06:02.972310 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.972314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.975963 (+ 3649us) server_negotiation.cc:407] Connection header received
0810 23:06:02.976128 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.976131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.976181 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.976229 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.977282 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37104: BlockingRecv error: recv got EOF from 127.0.0.1:37104 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 23:06:02.982420 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:02.980422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.15.56.61:33269)
0810 23:06:02.980539 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.980543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.981307 (+ 764us) server_negotiation.cc:407] Connection header received
0810 23:06:02.981558 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.981561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.981645 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.981697 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.982305 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44466: BlockingRecv error: recv got EOF from 127.0.0.1:44466 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 23:06:02.987792 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:02.986196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.15.56.61:33269)
0810 23:06:02.986316 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.986320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.986790 (+ 470us) server_negotiation.cc:407] Connection header received
0810 23:06:02.986978 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.986981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.987031 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.987079 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.987606 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44478: BlockingRecv error: recv got EOF from 127.0.0.1:44478 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 23:06:02.996881 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:02.992032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.15.56.61:33269)
0810 23:06:02.992126 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:02.992129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:02.995976 (+ 3847us) server_negotiation.cc:407] Connection header received
0810 23:06:02.996157 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:02.996160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:02.996221 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:02.996270 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:02.996777 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44486: BlockingRecv error: recv got EOF from 127.0.0.1:44486 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 23:06:03.004621 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:03.003064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44500 (local address 127.15.56.61:33269)
0810 23:06:03.003219 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:03.003223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:03.003720 (+ 497us) server_negotiation.cc:407] Connection header received
0810 23:06:03.003916 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:03.003919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:03.003968 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:03.004015 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:03.004516 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44500: BlockingRecv error: recv got EOF from 127.0.0.1:44500 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 23:06:03.016780 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:03.015074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.15.56.61:33269)
0810 23:06:03.015210 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:03.015214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:03.015774 (+ 560us) server_negotiation.cc:407] Connection header received
0810 23:06:03.015985 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:03.015989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:03.016042 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:03.016095 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:03.016661 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44514: BlockingRecv error: recv got EOF from 127.0.0.1:44514 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 23:06:03.037083 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:03.035581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.15.56.61:33269)
0810 23:06:03.035784 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:03.035790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:03.036097 (+ 307us) server_negotiation.cc:407] Connection header received
0810 23:06:03.036253 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:03.036257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:03.036326 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:03.036425 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:03.036959 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44526: BlockingRecv error: recv got EOF from 127.0.0.1:44526 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 23:06:03.072139 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:03.071003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.15.56.61:33269)
0810 23:06:03.071139 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:03.071144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:03.071286 (+ 142us) server_negotiation.cc:407] Connection header received
0810 23:06:03.071417 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:03.071421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:03.071475 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:03.071573 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:03.072012 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44532: BlockingRecv error: recv got EOF from 127.0.0.1:44532 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 23:06:03.139582 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:03.138328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.15.56.61:33269)
0810 23:06:03.138522 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:03.138527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:03.138633 (+ 106us) server_negotiation.cc:407] Connection header received
0810 23:06:03.138803 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:03.138806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:03.138859 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:03.138929 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:03.139455 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44546: BlockingRecv error: recv got EOF from 127.0.0.1:44546 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 23:06:03.271373 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:03.270183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.15.56.61:33269)
0810 23:06:03.270380 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:03.270384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:03.270513 (+ 129us) server_negotiation.cc:407] Connection header received
0810 23:06:03.270604 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:03.270607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:03.270661 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:03.270748 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:03.271241 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44552: BlockingRecv error: recv got EOF from 127.0.0.1:44552 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260810 23:06:03.530882 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:03.529745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.15.56.61:33269)
0810 23:06:03.529911 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:03.529915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:03.529934 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:06:03.530027 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:03.530030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:03.530079 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:03.530160 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:03.530739 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44554: BlockingRecv error: recv got EOF from 127.0.0.1:44554 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:06:03.863313 19132 catalog_manager.cc:1259] Loaded cluster ID: c870f4392c074aae8bda97590ccd8c72
I20260810 23:06:03.863412 19132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e: loading cluster ID for follower catalog manager: success
I20260810 23:06:03.865298 19132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e: acquiring CA information for follower catalog manager: success
I20260810 23:06:03.866657 19132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d394f0d9b5dc40018697da603cd7614e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:06:03.874034 19155 catalog_manager.cc:1259] Loaded cluster ID: c870f4392c074aae8bda97590ccd8c72
I20260810 23:06:03.874125 19155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6: loading cluster ID for follower catalog manager: success
I20260810 23:06:03.875953 19155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6: acquiring CA information for follower catalog manager: success
I20260810 23:06:03.877213 19155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f7eaedb09434e18b44a92efc80487c6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:06:04.049183 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:04.045119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.15.56.61:33269)
0810 23:06:04.045391 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:04.045395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:04.045493 (+ 98us) server_negotiation.cc:407] Connection header received
0810 23:06:04.045615 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:04.045617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:04.045765 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:04.045883 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:04.046100 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:04.046684 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:04.047554 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:04.047800 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:04.047896 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:04.048243 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:04.048779 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:04.048920 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:04.048954 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 23:06:04.049016 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":70,"threads_started":1}
I20260810 23:06:04.055987 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:04.052272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.15.56.60:34045)
0810 23:06:04.052499 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:04.052502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:04.052517 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:06:04.052660 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:04.052662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:04.052790 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:04.052899 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:04.053128 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:04.053658 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:04.054470 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:04.054691 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:04.054763 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:04.055070 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:04.055527 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:04.055620 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:04.055794 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 23:06:04.055856 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260810 23:06:04.064985 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:04.059138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39606 (local address 127.15.56.62:46615)
0810 23:06:04.059389 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:04.059393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:04.059599 (+ 206us) server_negotiation.cc:407] Connection header received
0810 23:06:04.060560 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:04.060564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:04.060726 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:04.060852 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:04.061196 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:04.061737 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:04.062764 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:04.063035 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:04.063201 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:04.063882 (+ 681us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:04.064212 (+ 330us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:04.064274 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:04.064723 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 23:06:04.064795 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260810 23:06:04.068640 19281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:04.060010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.15.56.60:34045)
0810 23:06:04.064724 (+ 4714us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:04.064730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:04.064743 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:06:04.064796 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:04.064799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:04.064848 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:04.064925 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:04.065394 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:04.066136 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:04.067358 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:04.067713 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:04.067821 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:04.068110 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:04.068192 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:04.068254 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:04.068413 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 23:06:04.068485 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4609}
I20260810 23:06:04.069108 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 18668
I20260810 23:06:04.074343 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:04.065144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.15.56.61:33269)
0810 23:06:04.065615 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:04.065620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:04.065636 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:04.065690 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:04.065693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:04.065746 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:04.065826 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:04.066150 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:04.067145 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:04.068280 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:04.068604 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:04.068692 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:04.068999 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:04.069088 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:04.069163 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:04.074120 (+ 4957us) server_negotiation.cc:299] Negotiation successful
0810 23:06:04.074194 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321}
I20260810 23:06:04.082942 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 18812
I20260810 23:06:04.101119 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 18951
2026-08-10T23:06:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3278 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:04 dist-test-slave-6bbx krb5kdc[19302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:04 dist-test-slave-6bbx krb5kdc[19302](info): set up 2 sockets
Aug 10 23:06:04 dist-test-slave-6bbx krb5kdc[19302](info): commencing operation
krb5kdc: starting...
W20260810 23:06:05.314568 15584 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/test-user.keytab.
Aug 10 23:06:05 dist-test-slave-6bbx krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.56.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
Entry for principal kudu/127.15.56.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab.
I20260810 23:06:05.455750 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43491
--webserver_interface=127.15.56.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:43491,127.15.56.61:36367,127.15.56.60:32851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:05.577951 19324 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:06:05.578277 19324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:05.578316 19324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:05.578423 19324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:05.582562 19324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:05.582641 19324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:05.582669 19324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:05.582690 19324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:05.582711 19324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:05.588369 19324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:43491,127.15.56.61:36367,127.15.56.60:32851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.62.keytab
--principal=kudu/127.15.56.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.62:43491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.56.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:05.589643 19324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:05.590759 19324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:05.600297 19330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:05.600358 19324 server_base.cc:1034] running on GCE node
W20260810 23:06:05.600302 19334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:05.600302 19329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:05.601225 19324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:05.602506 19324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:05.603725 19324 hybrid_clock.cc:648] HybridClock initialized: now 1786403165603701 us; error 32 us; skew 500 ppm
Aug 10 23:06:05 dist-test-slave-6bbx krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:05.609858 19324 init.cc:377] Logged in from keytab as kudu/127.15.56.62@KRBTEST.COM (short username kudu)
I20260810 23:06:05.610977 19324 webserver.cc:460] Webserver started at http://127.15.56.62:44463/ using document root <none> and password file <none>
I20260810 23:06:05.611563 19324 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:05.611629 19324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:05.611874 19324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:05.614092 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/data/instance:
uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b"
format_stamp: "Formatted at 2026-08-10 23:06:05 on dist-test-slave-6bbx"
server_key: "1b3d7921c37286c1ba2d8109f8b50c67"
server_key_iv: "5a707a52fe23e81fde2d71a6176c1a15"
server_key_version: "encryptionkey@0"
I20260810 23:06:05.614637 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/wal/instance:
uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b"
format_stamp: "Formatted at 2026-08-10 23:06:05 on dist-test-slave-6bbx"
server_key: "1b3d7921c37286c1ba2d8109f8b50c67"
server_key_iv: "5a707a52fe23e81fde2d71a6176c1a15"
server_key_version: "encryptionkey@0"
I20260810 23:06:05.618613 19324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:06:05.621167 19342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:05.622462 19324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:06:05.622578 19324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/wal
uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b"
format_stamp: "Formatted at 2026-08-10 23:06:05 on dist-test-slave-6bbx"
server_key: "1b3d7921c37286c1ba2d8109f8b50c67"
server_key_iv: "5a707a52fe23e81fde2d71a6176c1a15"
server_key_version: "encryptionkey@0"
I20260810 23:06:05.622669 19324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:05.649360 19324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:05.652837 19324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:05.653016 19324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:05.660290 19324 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.62:43491
I20260810 23:06:05.660584 19433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.62:43491 every 8 connection(s)
I20260810 23:06:05.661355 19324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:05.664438 19436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:05.664880 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 19324
I20260810 23:06:05.664983 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-0/wal/instance
I20260810 23:06:05.665261 15584 external_mini_cluster.cc:1428] Setting key 3117530be958aceb9007ab23d29f264d
I20260810 23:06:05.668606 19436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.56.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.61@KRBTEST.COM" created.
Aug 10 23:06:05 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:06:05.684715 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.671032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43491 (local address 127.0.0.1:48890)
0810 23:06:05.671571 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.671619 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:05.672591 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:05.673307 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:05.673313 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:05.673768 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:05.674034 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:05.674050 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.675246 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.675249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:05.676107 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:05.676119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.676780 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.676783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:05.676862 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:05.677386 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:05.677404 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:05.679238 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:05.681292 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:05.681303 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:05.681316 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:05.681715 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:05.682137 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:05.682140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:05.682142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:05.682295 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:05.683144 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:05.683151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:05.683317 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 23:06:05.683472 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 23:06:05.683823 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":128,"threads_started":1}
I20260810 23:06:05.684715 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.671927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.15.56.62:43491)
0810 23:06:05.672244 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:05.672255 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:05.672302 (+ 47us) server_negotiation.cc:407] Connection header received
0810 23:06:05.672905 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:05.672910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:05.673115 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:05.673291 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:05.674185 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.675074 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.676223 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.676631 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.676755 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:05.679433 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:05.679441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:05.679458 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:05.679486 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:05.681177 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:05.681844 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:05.681852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:05.681853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:05.682003 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:05.682405 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:05.682409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:05.682410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:05.683007 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:05.683165 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:05.683599 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 23:06:05.683953 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":202,"thread_start_us":102,"threads_started":1}
I20260810 23:06:05.687620 19436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:05.688493 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.688180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:36367 (local address 127.0.0.1:45028)
0810 23:06:05.688329 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.688418 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 23:06:05.688822 19343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.56.61:36367: connect: Connection refused (error 111)
I20260810 23:06:05.689841 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.689534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:36367 (local address 127.0.0.1:45038)
0810 23:06:05.689662 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.689721 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 23:06:05.689997 19436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.61:36367: Network error: Client connection negotiation failed: client connection to 127.15.56.61:36367: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab.
Entry for principal kudu/127.15.56.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab.
I20260810 23:06:05.696483 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.61:36367
--webserver_interface=127.15.56.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab
--principal=kudu/127.15.56.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:43491,127.15.56.61:36367,127.15.56.60:32851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:05.739287 19436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } attempt: 1
I20260810 23:06:05.740325 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.739958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:36367 (local address 127.0.0.1:45044)
0810 23:06:05.740142 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.740224 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 23:06:05.741127 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.740889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:36367 (local address 127.0.0.1:45060)
0810 23:06:05.740996 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.741054 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 23:06:05.741339 19436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.61:36367: Network error: Client connection negotiation failed: client connection to 127.15.56.61:36367: connect: Connection refused (error 111)
I20260810 23:06:05.809545 19436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } attempt: 2
I20260810 23:06:05.810544 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.810113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:36367 (local address 127.0.0.1:45064)
0810 23:06:05.810311 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.810436 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 23:06:05.811491 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.811238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:36367 (local address 127.0.0.1:45070)
0810 23:06:05.811346 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.811409 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.61:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 23:06:05.811689 19436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.61:36367: Network error: Client connection negotiation failed: client connection to 127.15.56.61:36367: connect: Connection refused (error 111)
W20260810 23:06:05.818043 19450 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:06:05.818367 19450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:05.818390 19450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:05.818435 19450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:05.822933 19450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:05.823004 19450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:05.823024 19450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:05.823040 19450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:05.823053 19450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:05.828785 19450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:43491,127.15.56.61:36367,127.15.56.60:32851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.61.keytab
--principal=kudu/127.15.56.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.61:36367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.56.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:05.829959 19450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:05.831017 19450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:05.840282 19455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:05.840288 19456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:05.840283 19459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:05.840862 19450 server_base.cc:1034] running on GCE node
I20260810 23:06:05.841526 19450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:05.842772 19450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:05.843941 19450 hybrid_clock.cc:648] HybridClock initialized: now 1786403165843914 us; error 33 us; skew 500 ppm
Aug 10 23:06:05 dist-test-slave-6bbx krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:05.846772 19450 init.cc:377] Logged in from keytab as kudu/127.15.56.61@KRBTEST.COM (short username kudu)
I20260810 23:06:05.847765 19450 webserver.cc:460] Webserver started at http://127.15.56.61:45345/ using document root <none> and password file <none>
I20260810 23:06:05.848373 19450 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:05.848448 19450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:05.848675 19450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:05.850836 19450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/data/instance:
uuid: "e51a3e019cab4c2b9f153d623d617daa"
format_stamp: "Formatted at 2026-08-10 23:06:05 on dist-test-slave-6bbx"
server_key: "465de35a24bcd1a517107b3aeba84dd1"
server_key_iv: "385f8a63d45fd3f7219583af75c6e906"
server_key_version: "encryptionkey@0"
I20260810 23:06:05.851421 19450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/wal/instance:
uuid: "e51a3e019cab4c2b9f153d623d617daa"
format_stamp: "Formatted at 2026-08-10 23:06:05 on dist-test-slave-6bbx"
server_key: "465de35a24bcd1a517107b3aeba84dd1"
server_key_iv: "385f8a63d45fd3f7219583af75c6e906"
server_key_version: "encryptionkey@0"
I20260810 23:06:05.855307 19450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 23:06:05.857936 19468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:05.859234 19450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:06:05.859380 19450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/wal
uuid: "e51a3e019cab4c2b9f153d623d617daa"
format_stamp: "Formatted at 2026-08-10 23:06:05 on dist-test-slave-6bbx"
server_key: "465de35a24bcd1a517107b3aeba84dd1"
server_key_iv: "385f8a63d45fd3f7219583af75c6e906"
server_key_version: "encryptionkey@0"
I20260810 23:06:05.859483 19450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:05.883846 19450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:05.887555 19450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:05.887812 19450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:05.896661 19450 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.61:36367
I20260810 23:06:05.896952 19570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.61:36367 every 8 connection(s)
I20260810 23:06:05.897831 19450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/data/info.pb
I20260810 23:06:05.901062 19571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:05.902859 19436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } attempt: 3
I20260810 23:06:05.905020 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 19450
I20260810 23:06:05.905134 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-1/wal/instance
I20260810 23:06:05.905400 15584 external_mini_cluster.cc:1428] Setting key 6c77c9700e96fb8f3d3a5110c18267fb
I20260810 23:06:05.907336 19571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:06:05 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.56.60@KRBTEST.COM; defaulting to no policy
I20260810 23:06:05.916790 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.903454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:36367 (local address 127.0.0.1:45078)
0810 23:06:05.903627 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.903645 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:05.903769 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:05.906163 (+ 2394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:05.906167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:05.906382 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:05.906523 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:05.906532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.908102 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.908106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:05.908916 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:05.908929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.909878 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.909881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:05.909995 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:05.910541 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:05.910554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:05.912229 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:05.914626 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:05.914630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:05.914632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:05.914895 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:05.915279 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:05.915283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:05.915285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:05.915390 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:05.916247 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:05.916251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:05.916361 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 23:06:05.916455 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 23:06:05.916612 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.56.60@KRBTEST.COM" created.
I20260810 23:06:05.918020 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.904208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45078 (local address 127.15.56.61:36367)
0810 23:06:05.904684 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:05.904698 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:05.904738 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:06:05.905467 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:05.905494 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:05.905875 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:05.906208 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:05.906662 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.907926 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.909054 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.909750 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.909918 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:05.912452 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:05.912458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:05.912471 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:05.912504 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:05.914511 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:05.915027 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:05.915035 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:05.915042 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:05.915181 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:05.915494 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:05.915497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:05.915499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:05.916135 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:05.916264 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:05.916635 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 23:06:05.916963 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":128,"threads_started":1}
Aug 10 23:06:05 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:06:05.920684 19436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:05.921425 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.921176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52534)
0810 23:06:05.921303 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.921355 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 23:06:05.922225 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.921914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52540)
0810 23:06:05.922086 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.922155 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260810 23:06:05.922377 19436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:32851: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
I20260810 23:06:05.922905 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.909482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.15.56.62:43491)
0810 23:06:05.909605 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:05.909610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:05.910466 (+ 856us) server_negotiation.cc:407] Connection header received
0810 23:06:05.910641 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:05.910644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:05.910692 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:05.910775 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:05.911643 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.912762 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.913971 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.914365 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.914484 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:05.918743 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:05.918748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:05.918750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:05.918779 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:05.920912 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:05.921458 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:05.921462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:05.921464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:05.921553 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:05.921900 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:05.921903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:05.921905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:05.922153 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:05.922243 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:05.922642 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 23:06:05.922762 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:06:05.922945 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.909408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43491 (local address 127.0.0.1:48906)
0810 23:06:05.910312 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.910373 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:05.910528 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:05.910948 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:05.910954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:05.911382 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:05.911519 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:05.911525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.912875 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.912879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:05.913817 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:05.913831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.914506 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.914510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:05.914649 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:05.915334 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:05.915348 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:05.918596 (+ 3248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:05.921029 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:05.921039 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:05.921045 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:05.921355 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:05.921674 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:05.921677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:05.921679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:05.921778 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:05.922263 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:05.922270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:05.922409 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 23:06:05.922560 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 23:06:05.922767 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":121,"threads_started":1}
I20260810 23:06:05.924387 19571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:06:05 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
I20260810 23:06:05.932494 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.924931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.15.56.61:36367)
0810 23:06:05.925020 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:05.925023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:05.925036 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:06:05.925148 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:05.925151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:05.925189 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:05.925250 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:05.925638 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.926218 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.927008 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.927289 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.927379 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:05.929659 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:05.929663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:05.929664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:05.929684 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:05.930873 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:05.931232 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:05.931235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:05.931237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:05.931308 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:05.931573 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:05.931576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:05.931578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:05.931800 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:05.931896 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:05.932185 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 23:06:05.932345 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":40}
I20260810 23:06:05.932494 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.924879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:36367 (local address 127.0.0.1:45094)
0810 23:06:05.924982 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.924994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:05.925072 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:05.925270 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:05.925273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:05.925448 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:05.925536 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:05.925541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.926314 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.926317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:05.926908 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:05.926917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:05.927392 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:05.927396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:05.927485 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:05.927907 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:05.927921 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:05.929539 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:05.930960 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:05.930963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:05.930965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:05.931149 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:05.931391 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:05.931394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:05.931395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:05.931493 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:05.931885 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:05.931888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:05.932001 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 23:06:05.932122 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:06:05.932283 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 23:06:05.933460 19571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:05.934238 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.933962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52544)
0810 23:06:05.934073 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.934159 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.56.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab.
Entry for principal kudu/127.15.56.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab.
W20260810 23:06:05.934893 19476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
I20260810 23:06:05.935408 15584 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
/tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.60:32851
--webserver_interface=127.15.56.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab
--principal=kudu/127.15.56.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.56.20:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.56.62:43491,127.15.56.61:36367,127.15.56.60:32851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:05.935871 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.935613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52552)
0810 23:06:05.935761 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.935816 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260810 23:06:05.936003 19571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:32851: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
I20260810 23:06:05.953572 19436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } attempt: 1
I20260810 23:06:05.954524 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.954174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52560)
0810 23:06:05.954371 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.954437 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 23:06:05.955258 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.955067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52572)
0810 23:06:05.955154 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.955199 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 23:06:05.955406 19436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:32851: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
I20260810 23:06:05.985276 19571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } attempt: 1
I20260810 23:06:05.986200 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.985876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52580)
0810 23:06:05.986050 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.986118 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 23:06:05.986941 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.986711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52596)
0810 23:06:05.986822 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.986882 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:06:05.987053 19571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:32851: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
I20260810 23:06:06.030573 19436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } attempt: 2
I20260810 23:06:06.031447 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.031107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52600)
0810 23:06:06.031278 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.031361 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 23:06:06.032222 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.032019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52614)
0810 23:06:06.032112 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.032158 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 23:06:06.032364 19436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:32851: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
W20260810 23:06:06.051339 19584 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:06:06.051762 19584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:06.051816 19584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:06.051880 19584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:06:06.055233 19571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } attempt: 2
W20260810 23:06:06.056147 19584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260810 23:06:06.056190 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.055847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52628)
0810 23:06:06.056035 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.056104 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 23:06:06.056211 19584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:06.056238 19584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:06.056263 19584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:06.056293 19584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:06.056922 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.056688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52640)
0810 23:06:06.056801 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.056857 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:06:06.057045 19571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.56.60:32851: Network error: Client connection negotiation failed: client connection to 127.15.56.60:32851: connect: Connection refused (error 111)
I20260810 23:06:06.062207 19584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.56.20:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.56.62:43491,127.15.56.61:36367,127.15.56.60:32851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/krb5kdc/kudu_127.15.56.60.keytab
--principal=kudu/127.15.56.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.56.60:32851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.56.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:06:06.063433 19584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:06.064582 19584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:06.073449 19593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:06.073455 19591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:06.073455 19598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:06.073879 19584 server_base.cc:1034] running on GCE node
I20260810 23:06:06.074550 19584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:06.075804 19584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:06.076967 19584 hybrid_clock.cc:648] HybridClock initialized: now 1786403166076941 us; error 48 us; skew 500 ppm
Aug 10 23:06:06 dist-test-slave-6bbx krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403166, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:06.079921 19584 init.cc:377] Logged in from keytab as kudu/127.15.56.60@KRBTEST.COM (short username kudu)
I20260810 23:06:06.080839 19584 webserver.cc:460] Webserver started at http://127.15.56.60:41417/ using document root <none> and password file <none>
I20260810 23:06:06.081408 19584 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:06.081470 19584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:06.081693 19584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:06.084004 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/data/instance:
uuid: "f15fba466a984ef9bb9618dc4f51221a"
format_stamp: "Formatted at 2026-08-10 23:06:06 on dist-test-slave-6bbx"
server_key: "e50b0429475ae37de61fa7240b70fce3"
server_key_iv: "b2594eab6e55db2c8872dbe5e1e9a052"
server_key_version: "encryptionkey@0"
I20260810 23:06:06.084569 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/wal/instance:
uuid: "f15fba466a984ef9bb9618dc4f51221a"
format_stamp: "Formatted at 2026-08-10 23:06:06 on dist-test-slave-6bbx"
server_key: "e50b0429475ae37de61fa7240b70fce3"
server_key_iv: "b2594eab6e55db2c8872dbe5e1e9a052"
server_key_version: "encryptionkey@0"
I20260810 23:06:06.088543 19584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 23:06:06.091017 19616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:06.092473 19584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:06:06.092597 19584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/data,/tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/wal
uuid: "f15fba466a984ef9bb9618dc4f51221a"
format_stamp: "Formatted at 2026-08-10 23:06:06 on dist-test-slave-6bbx"
server_key: "e50b0429475ae37de61fa7240b70fce3"
server_key_iv: "b2594eab6e55db2c8872dbe5e1e9a052"
server_key_version: "encryptionkey@0"
I20260810 23:06:06.092703 19584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:06.113703 19584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:06.117276 19584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:06.117458 19584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:06.124424 19584 rpc_server.cc:307] RPC server started. Bound to: 127.15.56.60:32851
I20260810 23:06:06.124682 19709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.56.60:32851 every 8 connection(s)
I20260810 23:06:06.125447 19584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/data/info.pb
I20260810 23:06:06.128530 19711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:06.131520 19436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } attempt: 3
I20260810 23:06:06.132726 19711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:06.133101 15584 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu as pid 19584
I20260810 23:06:06.133257 15584 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWzUZ3m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402867637343-15584-0/minicluster-data/master-2/wal/instance
I20260810 23:06:06.133589 15584 external_mini_cluster.cc:1428] Setting key cf212e036d70c957cc358d0e215ad6c9
Aug 10 23:06:06 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.62@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
Aug 10 23:06:06 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
Aug 10 23:06:06 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403166, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:06:06.145517 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.132081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52644)
0810 23:06:06.132243 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.132260 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:06.132390 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:06.135356 (+ 2966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:06.135359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:06.135573 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:06.135701 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:06.135711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.137204 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.137208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:06.137948 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:06.137959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.138802 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.138806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:06.139071 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.139552 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:06.139567 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:06.140977 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:06.143329 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:06.143333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:06.143336 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:06.143580 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:06.144043 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:06.144046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:06.144048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:06.144133 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:06.144906 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:06.144909 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:06.145024 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 23:06:06.145160 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 23:06:06.145334 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":151}
I20260810 23:06:06.145922 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.134831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.15.56.62:43491)
0810 23:06:06.134948 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.134952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.135229 (+ 277us) server_negotiation.cc:407] Connection header received
0810 23:06:06.135427 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.135430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.135474 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.135522 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.136552 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.137304 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.138691 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.139034 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.139118 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.142289 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:06.142293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:06.142296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:06.142327 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:06.143909 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.144472 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.144475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.144477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.144556 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.144863 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.144867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.144869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.145108 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:06.145195 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:06.145643 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 23:06:06.145769 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":54}
I20260810 23:06:06.146816 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.134317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.15.56.60:32851)
0810 23:06:06.134645 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.134656 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.134787 (+ 131us) server_negotiation.cc:407] Connection header received
0810 23:06:06.135162 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.135165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.135209 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.135366 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.136473 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.137305 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.138169 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.138661 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.138815 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.141810 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:06.141815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:06.141817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:06.141847 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:06.144040 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.144514 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.144517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.144519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.144601 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.144889 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.144893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.144894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.145091 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:06.145272 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:06.145648 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 23:06:06.145779 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":265,"thread_start_us":143,"threads_started":1}
I20260810 23:06:06.146800 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.134734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.62:43491 (local address 127.0.0.1:48908)
0810 23:06:06.135074 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.135137 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:06.135331 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:06.135741 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:06.135747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:06.136222 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:06.136400 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:06.136413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.137423 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.137426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:06.138551 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:06.138562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.139282 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.139288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:06.139391 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.139891 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:06.139912 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:06.142134 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:06.144040 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:06.144050 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:06.144058 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:06.144357 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:06.144665 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:06.144668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:06.144670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:06.144758 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:06.145248 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:06.145255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:06.145384 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 23:06:06.145974 (+ 590us) client_negotiation.cc:240] Negotiation successful
0810 23:06:06.146486 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":54,"thread_start_us":92,"threads_started":1}
I20260810 23:06:06.146800 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.133049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.15.56.60:32851)
0810 23:06:06.133600 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.133647 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.133697 (+ 50us) server_negotiation.cc:407] Connection header received
0810 23:06:06.134587 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.134613 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.135089 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.135808 (+ 719us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.135913 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.137092 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.138076 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.138667 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.138921 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.141120 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:06.141127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:06.141141 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:06.141168 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:06.143201 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.143771 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.143778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.143786 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.143934 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.144217 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.144221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.144222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.144819 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:06.144942 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:06.145350 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 23:06:06.145692 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":369,"thread_start_us":179,"threads_started":1}
I20260810 23:06:06.148202 19571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } attempt: 3
I20260810 23:06:06.149523 19711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:06.152339 19436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b: Bootstrap starting.
Aug 10 23:06:06 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403166, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
I20260810 23:06:06.155504 19436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b: Neither blocks nor log segments found. Creating new log.
Aug 10 23:06:06 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.61@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
Aug 10 23:06:06 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.62@KRBTEST.COM
I20260810 23:06:06.156551 19436 log.cc:824] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:06.158962 19436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b: No bootstrap required, opened a new log
I20260810 23:06:06.160037 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.150050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.15.56.61:36367)
0810 23:06:06.150168 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.150172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.150217 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:06:06.150374 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.150378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.150422 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.150469 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.151227 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.151950 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.153030 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.153352 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.153425 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.156104 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:06.156108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:06.156110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:06.156138 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:06.157851 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.158418 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.158422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.158424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.158514 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.158865 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.158869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.158871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.159187 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:06.159268 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:06.159734 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 23:06:06.159869 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":35}
I20260810 23:06:06.160236 19576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.148678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52658)
0810 23:06:06.148826 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.148842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:06.148944 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:06.150244 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:06.150247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:06.150442 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:06.150609 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:06.150616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.151555 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.151558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:06.152360 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:06.152372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.153105 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.153109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:06.153198 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.154101 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:06.154116 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:06.156780 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:06.158604 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:06.158607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:06.158609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:06.158851 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:06.159181 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:06.159184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:06.159186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:06.159277 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:06.159750 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:06.159753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:06.159850 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 23:06:06.159935 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:06:06.160073 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":35}
I20260810 23:06:06.160916 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.149305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.15.56.60:32851)
0810 23:06:06.149426 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.149430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.149443 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:06:06.149859 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.149862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.150115 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.150552 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.150697 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.151435 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.152556 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.152986 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.154861 (+ 1875us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.156934 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:06.156938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:06.156940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:06.156964 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:06.158474 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.158977 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.158980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.158981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.159064 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.159401 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.159403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.159405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.159604 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:06.160524 (+ 920us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:06.160602 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 23:06:06.160727 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":47}
I20260810 23:06:06.161219 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.149974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.61:36367 (local address 127.0.0.1:45108)
0810 23:06:06.150133 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.150146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:06.150283 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:06.150769 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:06.150771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:06.150954 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:06.151069 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:06.151077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.152092 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.152094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:06.152893 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:06.152905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.153583 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.153587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:06.153689 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.154104 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:06.154118 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:06.155965 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:06.157972 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:06.157976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:06.157978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:06.158300 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:06.158646 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:06.158649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:06.158651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:06.158752 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:06.159393 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:06.159396 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:06.159501 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 23:06:06.160656 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0810 23:06:06.160804 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":27}
I20260810 23:06:06.161293 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.149696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.15.56.62:43491)
0810 23:06:06.149801 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.149804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.149959 (+ 155us) server_negotiation.cc:407] Connection header received
0810 23:06:06.150245 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.150248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.150297 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.150347 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.151190 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.151900 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.152956 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.153265 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.153339 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.157533 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:06.157538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:06.157541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:06.157569 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:06.159365 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.159870 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.159873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.159874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.159954 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.160279 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.160283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.160284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.160499 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:06.160642 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:06.161024 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 23:06:06.161161 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:06:06.162518 19711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:06:06.163553 19436 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } }
I20260810 23:06:06.163834 19436 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbed5c8c3bdb470ab11a22f74cae8a6b, State: Initialized, Role: FOLLOWER
I20260810 23:06:06.164178 19436 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } }
I20260810 23:06:06.164243 19571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa: Bootstrap starting.
I20260810 23:06:06.165400 19721 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } } }
I20260810 23:06:06.165628 19436 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:06.165647 19721 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:06.167109 19571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa: Neither blocks nor log segments found. Creating new log.
Aug 10 23:06:06 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403166, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.56.60@KRBTEST.COM for kudu/127.15.56.60@KRBTEST.COM
I20260810 23:06:06.168589 19571 log.cc:824] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:06.171636 19571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa: No bootstrap required, opened a new log
I20260810 23:06:06.172106 19718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.163007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.56.60:32851 (local address 127.0.0.1:52668)
0810 23:06:06.163341 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.163352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:06.163417 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:06.163640 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:06.163643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:06.163831 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:06.163918 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:06.163925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.164744 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.164747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:06.165265 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:06.165273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.165911 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.165914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:06.165974 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.166386 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:06.166399 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:06.167897 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:06.170319 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:06.170322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:06.170324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:06.170558 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:06.170839 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:06.170841 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:06.170843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:06.170914 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:06.171566 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:06.171569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:06.171702 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 23:06:06.171809 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:06:06.171935 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280}
I20260810 23:06:06.172514 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.163077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.15.56.60:32851)
0810 23:06:06.163183 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.163187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.163406 (+ 219us) server_negotiation.cc:407] Connection header received
0810 23:06:06.163509 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.163512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.163549 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.163611 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.164024 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.164649 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.165373 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.165697 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.165807 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.168026 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:06.168029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:06.168032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:06.168056 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:06.169598 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.170669 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.170672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.170673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.170755 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.170994 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.170996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.170998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.171193 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:06.171526 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:06.172286 (+ 760us) server_negotiation.cc:299] Negotiation successful
0810 23:06:06.172388 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 23:06:06 dist-test-slave-6bbx krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.56.61@KRBTEST.COM
I20260810 23:06:06.171316 19324 master_runner.cc:186] Error getting master registration for 127.15.56.61:36367: OK, instance_id { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" instance_seqno: 1786403165894329 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:06:06.175078 19324 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:06:06.175572 19571 raft_consensus.cc:348] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } }
I20260810 23:06:06.175846 19571 raft_consensus.cc:729] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51a3e019cab4c2b9f153d623d617daa, State: Initialized, Role: FOLLOWER
I20260810 23:06:06.176154 19571 consensus_queue.cc:260] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } }
I20260810 23:06:06.176530 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.167777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.15.56.61:36367)
0810 23:06:06.167892 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.167896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.167909 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:06:06.167944 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.167947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.167991 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.168075 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.168819 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.169479 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.170488 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.170791 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.170881 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.173626 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:06.173630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:06.173633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:06.173657 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:06.174884 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.175329 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.175332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.175333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.175423 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.175689 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.175693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.175695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.175896 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:06.176083 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:06.176286 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 23:06:06.176401 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:06:06.177579 19571 sys_catalog.cc:564] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:06.178030 19711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a: Bootstrap starting.
I20260810 23:06:06.179844 15584 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:06:06.180712 19711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:06.181053 19385 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48916
I20260810 23:06:06.181280 19385 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:06:06.181382 19385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:06.181599 19711 log.cc:824] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a: Log is configured to *not* fsync() on all Append() calls
W20260810 23:06:06.181803 19739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:06.181865 19739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:06:06.182512 19763 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:06.182578 19763 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:06.182888 19385 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } }
I20260810 23:06:06.183914 19385 leader_election.cc:288] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [CANDIDATE]: Term 1 election: Requested vote from peers e51a3e019cab4c2b9f153d623d617daa (127.15.56.61:36367), f15fba466a984ef9bb9618dc4f51221a (127.15.56.60:32851)
I20260810 23:06:06.184370 19517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e51a3e019cab4c2b9f153d623d617daa"
I20260810 23:06:06.185949 19711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a: No bootstrap required, opened a new log
I20260810 23:06:06.189734 19711 raft_consensus.cc:348] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } }
I20260810 23:06:06.189944 19711 raft_consensus.cc:729] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15fba466a984ef9bb9618dc4f51221a, State: Initialized, Role: FOLLOWER
I20260810 23:06:06.190246 19711 consensus_queue.cc:260] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } }
I20260810 23:06:06.190680 19780 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } } }
I20260810 23:06:06.190863 19780 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:06.191485 19711 sys_catalog.cc:564] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:06.191843 19660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15fba466a984ef9bb9618dc4f51221a"
I20260810 23:06:06.192085 19660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:06.193390 19743 sys_catalog.cc:455] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } } }
I20260810 23:06:06.193516 19660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbed5c8c3bdb470ab11a22f74cae8a6b in term 1.
I20260810 23:06:06.193579 19743 sys_catalog.cc:458] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:06.193813 19517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:06.195592 19517 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbed5c8c3bdb470ab11a22f74cae8a6b in term 1.
I20260810 23:06:06.196269 19348 leader_election.cc:302] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbed5c8c3bdb470ab11a22f74cae8a6b, f15fba466a984ef9bb9618dc4f51221a; no voters:
I20260810 23:06:06.197109 19721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:06.197405 19721 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [term 1 LEADER]: Becoming Leader. State: Replica: cbed5c8c3bdb470ab11a22f74cae8a6b, State: Running, Role: LEADER
W20260810 23:06:06.197579 19798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:06:06.197662 19798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:06:06.197835 19721 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } }
I20260810 23:06:06.194118 19450 master_runner.cc:186] Error getting master registration for 127.15.56.60:32851: OK, instance_id { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" instance_seqno: 1786403166122506 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:06:06.200925 19450 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:06:06.202296 19800 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbed5c8c3bdb470ab11a22f74cae8a6b. Latest consensus state: current_term: 1 leader_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } } }
I20260810 23:06:06.202438 19800 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:06.205631 19584 master_runner.cc:263] No leader master found from master f15fba466a984ef9bb9618dc4f51221a
I20260810 23:06:06.205701 19584 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:06:06.208930 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.186393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.15.56.61:36367)
0810 23:06:06.186507 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.186511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.186526 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:06:06.186723 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.186727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.186774 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.186850 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.188322 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.189083 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.190980 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.191320 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.191399 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.205272 (+ 13873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:06.205276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:06.205279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:06.205308 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:06.206941 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.207449 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.207452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.207454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.207549 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.207900 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.207904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.207906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.208134 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:06.208241 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:06.208620 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 23:06:06.208765 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:06:06.209389 19812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:06.210451 19812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:06.214061 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.200441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.15.56.62:43491)
0810 23:06:06.200575 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.200579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.200595 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:06:06.200636 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.200639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.200685 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.200771 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.203435 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.204224 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.206424 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.206770 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.206850 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.208507 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:06.208511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:06.208514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:06.208544 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:06.211772 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.212326 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.212329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.212331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.212429 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.212761 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.212765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.212767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.213014 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:06.213139 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:06.213759 (+ 620us) server_negotiation.cc:299] Negotiation successful
0810 23:06:06.213894 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:06:06.216572 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.199886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.15.56.60:32851)
0810 23:06:06.200008 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.200013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.200028 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:06:06.200069 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.200072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.200117 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.200206 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.202736 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.203423 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.204509 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:06.204809 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:06.204891 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:06.212749 (+ 7858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:06.212754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:06.212757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:06.212791 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:06.214336 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.214834 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.214837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.214839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.214935 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:06.215229 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:06.215232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:06.215234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:06.215467 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:06.215605 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:06.216284 (+ 679us) server_negotiation.cc:299] Negotiation successful
0810 23:06:06.216403 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:06:06.227795 19660 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [term 1 FOLLOWER]: Refusing update from remote peer cbed5c8c3bdb470ab11a22f74cae8a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:06.227795 19517 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [term 1 FOLLOWER]: Refusing update from remote peer cbed5c8c3bdb470ab11a22f74cae8a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:06.228493 19800 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:06.229094 19800 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:06.233191 19743 sys_catalog.cc:455] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbed5c8c3bdb470ab11a22f74cae8a6b. Latest consensus state: current_term: 1 leader_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } } }
I20260810 23:06:06.233314 19743 sys_catalog.cc:458] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:06.233426 19780 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbed5c8c3bdb470ab11a22f74cae8a6b. Latest consensus state: current_term: 1 leader_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } } }
I20260810 23:06:06.233568 19780 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:06.243012 19800 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } } }
I20260810 23:06:06.243180 19800 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:06.243549 19814 mvcc.cc:205] Tried to move back new op lower bound from 7317107368843522048 to 7317107368750469120. Current Snapshot: MvccSnapshot[applied={T|T < 7317107368843522048}]
I20260810 23:06:06.244794 19743 sys_catalog.cc:455] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } } }
I20260810 23:06:06.244908 19743 sys_catalog.cc:458] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:06.247290 19819 mvcc.cc:205] Tried to move back new op lower bound from 7317107368843522048 to 7317107368750469120. Current Snapshot: MvccSnapshot[applied={T|T < 7317107368843522048}]
I20260810 23:06:06.248093 19721 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } } }
I20260810 23:06:06.248248 19721 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:06.249118 19780 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbed5c8c3bdb470ab11a22f74cae8a6b" member_type: VOTER last_known_addr { host: "127.15.56.62" port: 43491 } } peers { permanent_uuid: "e51a3e019cab4c2b9f153d623d617daa" member_type: VOTER last_known_addr { host: "127.15.56.61" port: 36367 } } peers { permanent_uuid: "f15fba466a984ef9bb9618dc4f51221a" member_type: VOTER last_known_addr { host: "127.15.56.60" port: 32851 } } }
I20260810 23:06:06.249248 19780 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:06:06.250430 19816 mvcc.cc:205] Tried to move back new op lower bound from 7317107368843522048 to 7317107368750469120. Current Snapshot: MvccSnapshot[applied={T|T < 7317107368843522048}]
I20260810 23:06:06.254829 19812 catalog_manager.cc:1347] Generated new cluster ID: 7186a3a70aa04ea3a98fa6f46ff82168
I20260810 23:06:06.254897 19812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:06.276965 19812 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:06.278345 19812 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:06.299386 19812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbed5c8c3bdb470ab11a22f74cae8a6b: Generated new TSK 0
I20260810 23:06:06.300443 19812 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:06:06.341152 19444 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.338572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.15.56.62:43491)
0810 23:06:06.338734 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.338738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.340016 (+ 1278us) server_negotiation.cc:407] Connection header received
0810 23:06:06.340219 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.340223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.340275 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.340326 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.341006 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48920: BlockingRecv error: recv got EOF from 127.0.0.1:48920 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 23:06:06.342614 19714 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.339382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.15.56.60:32851)
0810 23:06:06.339498 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.339503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.341688 (+ 2185us) server_negotiation.cc:407] Connection header received
0810 23:06:06.341868 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.341871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.341925 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.341977 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.342493 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52680: BlockingRecv error: recv got EOF from 127.0.0.1:52680 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 23:06:06.344767 19573 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.343247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.15.56.61:36367)
0810 23:06:06.343362 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.343367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.343903 (+ 536us) server_negotiation.cc:407] Connection header received
0810 23:06:06.343987 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.343990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.344039 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.344103 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.344631 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45120: BlockingRecv error: recv got EOF from 127.0.0.1:45120 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 23:06:06.349792 19573 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.348017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.15.56.61:36367)
0810 23:06:06.348125 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.348129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.348707 (+ 578us) server_negotiation.cc:407] Connection header received
0810 23:06:06.348943 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.348946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.349037 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.349092 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.349674 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45126: BlockingRecv error: recv got EOF from 127.0.0.1:45126 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 23:06:06.355365 19573 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.353768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.15.56.61:36367)
0810 23:06:06.353881 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.353885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.354408 (+ 523us) server_negotiation.cc:407] Connection header received
0810 23:06:06.354602 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.354605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.354673 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.354724 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.355251 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45132: BlockingRecv error: recv got EOF from 127.0.0.1:45132 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 23:06:06.361874 19573 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.360056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.15.56.61:36367)
0810 23:06:06.360232 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.360237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.360803 (+ 566us) server_negotiation.cc:407] Connection header received
0810 23:06:06.361030 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.361033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.361095 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.361153 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.361754 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45134: BlockingRecv error: recv got EOF from 127.0.0.1:45134 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 23:06:06.373466 19573 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.368808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.15.56.61:36367)
0810 23:06:06.368977 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.368982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.372191 (+ 3209us) server_negotiation.cc:407] Connection header received
0810 23:06:06.372474 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.372479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.372548 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.372612 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.373328 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45148: BlockingRecv error: recv got EOF from 127.0.0.1:45148 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 23:06:06.386334 19573 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.384711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45160 (local address 127.15.56.61:36367)
0810 23:06:06.384868 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.384873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.385237 (+ 364us) server_negotiation.cc:407] Connection header received
0810 23:06:06.385458 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.385462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.385521 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.385574 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.386201 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45160: BlockingRecv error: recv got EOF from 127.0.0.1:45160 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 23:06:06.406862 19573 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.405570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.15.56.61:36367)
0810 23:06:06.405704 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.405709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.405987 (+ 278us) server_negotiation.cc:407] Connection header received
0810 23:06:06.406119 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.406122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.406174 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.406244 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.406736 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45162: BlockingRecv error: recv got EOF from 127.0.0.1:45162 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 23:06:06.442566 19573 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.441247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.15.56.61:36367)
0810 23:06:06.441448 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.441452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.441487 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:06:06.441684 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.441688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.441761 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.441843 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.442430 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45174: BlockingRecv error: recv got EOF from 127.0.0.1:45174 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260810 23:06:06.509903 19573 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.508824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.15.56.61:36367)
0810 23:06:06.509026 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.509030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.509092 (+ 62us) server_negotiation.cc:407] Connection header received
0810 23:06:06.509223 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.509225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.509273 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.509347 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.509772 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45182: BlockingRecv error: recv got EOF from 127.0.0.1:45182 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260810 23:06:06.641208 19573 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.640071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45194 (local address 127.15.56.61:36367)
0810 23:06:06.640222 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.640227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.640378 (+ 151us) server_negotiation.cc:407] Connection header received
0810 23:06:06.640499 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.640502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.640554 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.640625 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.641078 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45194: BlockingRecv error: recv got EOF from 127.0.0.1:45194 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 23:06:06.900806 19573 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:06.899586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.15.56.61:36367)
0810 23:06:06.899755 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:06.899759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:06.899922 (+ 163us) server_negotiation.cc:407] Connection header received
0810 23:06:06.900046 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:06.900049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:06.900103 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:06.900178 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:06.900692 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45196: BlockingRecv error: recv got EOF from 127.0.0.1:45196 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:06:07.184023 19763 catalog_manager.cc:1259] Loaded cluster ID: 7186a3a70aa04ea3a98fa6f46ff82168
I20260810 23:06:07.184116 19763 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa: loading cluster ID for follower catalog manager: success
I20260810 23:06:07.185911 19763 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa: acquiring CA information for follower catalog manager: success
I20260810 23:06:07.187151 19763 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e51a3e019cab4c2b9f153d623d617daa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:06:07.199012 19798 catalog_manager.cc:1259] Loaded cluster ID: 7186a3a70aa04ea3a98fa6f46ff82168
I20260810 23:06:07.199079 19798 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a: loading cluster ID for follower catalog manager: success
I20260810 23:06:07.200775 19798 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a: acquiring CA information for follower catalog manager: success
I20260810 23:06:07.201957 19798 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f15fba466a984ef9bb9618dc4f51221a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:06:07.419329 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:07.415173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45212 (local address 127.15.56.61:36367)
0810 23:06:07.415506 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:07.415510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:07.415527 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:06:07.415654 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:07.415657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:07.415826 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:07.415963 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:07.416310 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:07.416895 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:07.417782 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:07.418007 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:07.418095 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:07.418424 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:07.418905 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:07.419025 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:07.419088 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 23:06:07.419151 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":116,"threads_started":1}
I20260810 23:06:07.426101 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:07.422400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.15.56.60:32851)
0810 23:06:07.422643 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:07.422648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:07.422663 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:06:07.422790 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:07.422793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:07.422923 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:07.423028 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:07.423261 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:07.423810 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:07.424626 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:07.424863 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:07.424950 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:07.425254 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:07.425699 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:07.425811 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:07.425882 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 23:06:07.425943 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260810 23:06:07.432848 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:07.429113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.15.56.61:36367)
0810 23:06:07.429257 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:07.429263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:07.429591 (+ 328us) server_negotiation.cc:407] Connection header received
0810 23:06:07.429770 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:07.429774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:07.429833 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:07.429897 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:07.430131 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:07.430679 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:07.431752 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:07.432081 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:07.432172 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:07.432397 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:07.432468 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:07.432522 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:07.432633 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 23:06:07.432711 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:06:07.433264 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:07.429464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.15.56.60:32851)
0810 23:06:07.429573 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:07.429577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:07.429802 (+ 225us) server_negotiation.cc:407] Connection header received
0810 23:06:07.429902 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:07.429904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:07.429947 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:07.430012 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:07.430254 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:07.431016 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:07.432129 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:07.432445 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:07.432532 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:07.432795 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:07.432871 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:07.432914 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:07.433081 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 23:06:07.433136 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 23:06:07.434211 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:07.429058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.15.56.62:43491)
0810 23:06:07.429363 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:07.429367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:07.429544 (+ 177us) server_negotiation.cc:407] Connection header received
0810 23:06:07.429655 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:07.429658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:07.429874 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:07.430244 (+ 370us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:07.430360 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:07.431127 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:07.432605 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:07.432916 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:07.433276 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:07.433430 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:07.433823 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:07.433911 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:07.434012 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:06:07.434066 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":101,"threads_started":1}
I20260810 23:06:07.437359 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 19324
I20260810 23:06:07.457626 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 19450
I20260810 23:06:07.471228 15584 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWzUZ3m/build/debug/bin/kudu with pid 19584
2026-08-10T23:06:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3374 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6653 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (299836 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST